I'm on the anti-astrology side, but this isn't a good counter any more than "I hate white rappers too, so my hating rap can't be racist" is. There is no contradiction in hating something associated with women on a misogynistic basis and also hating men who like it.