That too - same reason why racism allegories like don't work so well. When you try to write a story as 1:1 parallel to real life using species with vast inherent physical disparities, things tend to get messy.

That's not to say you can't draw upon the history of racism, colonialism, etc., for your scifi or fantasy ofc, but it requires a little more effort and consideration than that to make a compelling story