I've heard people argue the whole "POC and colored person are the same because they sound similar!" thing, but it completely neglects the fact that whether a term is used derogatorily or not is more important for considering it a slur than just "how it sounds." You'd seldom, if ever, hear people use POC as a slur, but colored person, like you said, has been used as such.