It's quite racist (and antisemetic) to think that just because things look similar that they were intended to be the same thing. They're not.

"Oh, you think that this fantasy race that has many of the anti-semitic tropes of Jewish people might in fact be a reference to Jewish people? Well, you're the one who associates those features with Jewish people, not me. Just saying."

This is where the moronic "maybe the curtains are just blue" reddit-tier analysis of literature gets you - completely unable to see any kind of allegory or metaphor, especially when bigots say that, no, that person in that book totally isn't a racist caricature, it's just a person with those traits!

If I wrote a book about a fantasy world where I used lots of sexist stereotypes about women - that they're less intelligent; that they're inherently subservient to men; that they "belong in the kitchen"; that they should be "barefoot and pregnant"; etc, and without ever even making a critical judgement of those traits or showed that the men in that society are bad for maintaining this status quo, then I would rightfully be called a raging sexist by people. They would probably believe I was one of those tradcath, alt-right MGTOW incel people. If I turned around and said "Uh, it says a lot about liberals that they think these traits are stereotypically true of women! Maybe they're the real sexists, not conservatives?" then you would, hopefully (though I'm not so sure given your lack of sensitivity towards Jewish people) call me a total fucking dipshit.