I usually see white people use it against people of color of Cuban descent, but never against white people with the exact same viewpoints who are white.

I've seen Ted Cruz, Marco Rubio, and Robert Menendez described as gusanos quite a few times.

I do see a trend in Terminally Online Media to have a Token Black Guy give the "I dream of a free and democratic Cuba!" line, and then a dozen other extremely generic accounts scream "Racism!" at anyone who shows disgust. But there is no shortage of pasty faced "Fidel took all my slaves" mfers who earn the label, regularly.