Sure, there are related non slur words, but those aren't applied to people. "To removed" is a verb, not an adjective or noun. "removedent" is an adjective, but describes the action of soemething, not a quality. Using the noun or past tense adjective is the slur.

And language doesn't have logic. Like, its pretty weird to suggest it does. Why is fuck a curse? Because its a curse. Why do we call blue things blue? Becabse they are blue. Language and logic rarely intersect (except in lojban). Language is determined by usage, and the usage of that word as a noun is as a slur.