This is the same issue the LGBT community suffers from, as well as people of color, people with accents, etc. People always find ways to make themselves feel superior to others by pointing out something different. You choosing to wear a mask just makes those kinds of people laugh because they either 1. Feel better than you for not wearing masks or 2. They are uneducated/ignorant and follow others' opinions without question.
I suppose the only way to turn it back around on them would be to try scaring them by saying you have something contagious like strep throat, or worse, so they'll shut up and leave you alone. Otherwise my only other advice would be to just do your best to not engage with these types of people.
Dominant personalities and work styles almost always make it up to management.