My thoughts is that it's a simple situation really. If they're harassing or assaulting people, the women will call the cops or something, simple situation and get the guy arrested. If he's not doing anything, it's nothing harmful. Apparently that's not a solid enough answer. What should I have said?
Is there no one in the replies here who thinks women have a legitimate discomfort, or unsafe feeling, having men around in a toilet space, even if the men aren't actively being harmful?
No women here who had difficult upbringings with men? No men whose daughter or sister or female friend feels uncomfortable letting certain barriers down around strange men?
Of course there is an important discussion about how bathroom culture changes as society's acceptance of trans people changes.
But, OP, I think what you would do best beyond what you said, is to acknowledge that some women have a legitimate concern, even if there's not an easy answer. Once you have that point of agreement - once the other person can see you care about the concern they're coming from - you have a foundation for discussing a real problem and/or solution.
Otherwise you're just buttimg heads to win, and asking an internet echo chamber to adjudicate.