My thoughts is that it's a simple situation really. If they're harassing or assaulting people, the women will call the cops or something, simple situation and get the guy arrested. If he's not doing anything, it's nothing harmful. Apparently that's not a solid enough answer. What should I have said?
Whatever you answer to a question like that is going to be attacked. The best way is not to answer, but reply with a different question, for instance "what are you afraid of?"