literally just title. If a society exists where there is a huge disparity in terms of social status and distribution of power, regardless of what gender has greater control, we should be opposed to that power dynamic. Like just swapping genders would literally just place men in exactly the same position that women are in now.
It would probably be a different position (because the material conditions that allow a matriarchy would be different) but yes, it would be bad and we would have to stop it.