"but there are some practical reasons that people don’t like having travelers in their communities." Would you say then that American conservatives are right when saying immigrants steal their jobs, so they shouldn't be allowed in? Or that having Black people in a neighbourhood raises crime? This stance is no different than saying something like that.

Also all cultures have issues in them. A lot of people of colour or working class people are homophobic/transphobic/seixst/etc. No culture is perfect. Just because a group may be sexist or have "childcare issues" (I would heavily fight against this idea as an actual gypsy) doesn't mean leftists should be allowed to be racist or find them "at odds" with leftism.