And it has all kinds of implications and context and shit.
How the fuck am I supposed to tell what's just people doing things at random and what's a part of all-singing-all-dancing-shit-show the neurotypicals call "social norms"? Down with NTs.
Honestly while it's really fucking annoying that refusal to participate could result in some silly neuronormo interpreting a completely innocuous action as an offense, and it can cut you off from social shit, I think it's basically the only way. The same button never does the same thing twice. Screeching rn.
Well it does bug me when I get alerted to shit like that plays into REJECTION SENSITIVITY
oh boy!!! Same as not making them my problem, that works great and it's fine for me until they cause a scene about it for whatever reason 
Yep, rejection sucks. Until you understand that they're the problem, not you, then "rejection" is just a good filter for yourself. They're filtering themselves out so you don't have to deal with their shittiness.
All the more when they cause a scene. I usually just walk away. Idk about you, how old you are, which social situation you're in, but it's interesting how much you can just walk away from that you never thought you could. Most of the time they either respect you more or don't try their shit anymore because they know it doesn't work on you.
Anyway, I wish you lots of happiness and good interactions instead :)