Given that during early stages of fetal development both sexes are present in everyone, too different dimorphism would not be advantageous for evolutionary survival.

Thats why for example the body “reuses” the material of a man’s tongue for his “head”.