Unless you're measuring people by say, every single measurement required to properly size a suit (hips, shoulders, ect) alongside things like muscle mass and weight then it's still difficult to properly group people.

At least all of the above is very relevant in sports like swimming, running, ect, although it would certainly make the sports more interesting with deeper classifications like this. A lot of people quit these sports because they don't have the right build.