Now I'm not an expert by any means, even though this is my own family history so I really should know a lot more; but my impression is that Scandinavians had privileges in the USA from being protestant

Early America has A LOT of anti-Catholic brainworms, the Puritans were constantly fretting about "Popish Conspiracies", there were tons of nativist anti-Catholic political parties and gangs. A lot of German immigrants were Catholic, and basically all Irish were, it was probably a bigger contribution towards hostility towards them than ethnic tensions.