“It’s a simple fact that men’s brains are just bigger than women’s.

Actually, it is true that men's brains are bigger than women's, it's just that brain size doesn't correlate with intelligence contrary to popular belief.

Similarly, men have a big biological advantage in certain sports. In some sports there isn't so much of a difference or women have an advantage (gymnastics, very long distance sports etc).

I'm not saying that to justify exclusion of trans people from their preferred categories (which shouldn't happen) but some people think complete gender desegregation of sports would be preferable and I think that's a bad solution because it would remove incentives for women to enter competitive sports and would ruin their chances at them. I know a number of women in sports close to Olympic level (mostly swimming) and I know they would never accept complete desegregation.

Opening women's sports up for trans women does not not seriously harm cis women because it's not like trans women are so many and have such a huge advantage that they're just completely overcrowding women's sports to the point where they exclude everyone. Like, alright, a trans woman won something once. Why is that supposed to be scary? But complete desegregation would practically mean exclusion of women from many sports and would amplify the societal differences that contribute (on top of the biological factor) to women having a disadvantage.