Well calling someone anxious isn't really a derogatory thing is it? It's describing an emotion that isn't seen as negative. Meanwhile, I've never seen anyone call someone self-absorbed without it being intended to critique/speak negatively about that action. It's why I specifically chose autism as my example, because there are a lot of issues of people out there using it pejoratively just as narcissism is used.