The heavy emphasis on abstraction before many minds are ready to think that way traumatizes a lot of people.

Math is also incoherently coded as something you can do if you're "smart". So when people struggle to get it they will rarely just accept that they aren't "smart" or just reject the bullshit idea that math is a measure of intelligence but instead the cognitive dissonance builds a potent resentment toward it that they carry around as adults.

There's a lot of adults who hold similar resentments toward PE, it's also experienced as an institutional psychic attack on your ego but instead of your inherent intelligence it's the validity of your body.