I don't know the exact reasoning for it, I would guess it's because '.' is also used to delimit file types "A book on the.pdf" < "A book on the pdf.pdf" or "Book.pdf" < "Book sequel.pdf"

I guess your confusion has in part to do with reading digits as numbers, but within a string they are part of an alphabet, every character stands on its own and has no relation to the characters around it. There is no difference between "5 1", "5.1" and "511" you just pick an ordering of the alphabet and then sort each character accordingly.