It's not selfish, you do not exist to serve anyone else or anything that you have built. If it's not good for you the way things are, changing yourself is entirely valid. If the people or structures or whatever cannot handle you, that is their problem. People who truly care for you will see that you are doing what's best for you.