It's a dumb argument because unless you're willing to lie to moderators with different usernames to trick them into not knowing who you are (ban evasion) being banned on one shitty instance locks you out of it and any communities it has. Lemmy.world has many of the bigger ones, lemmy.ml too.
Moving on isn't as easy as people make it seem. This is one of those Fediverse lies that people regurgitate ad-nauseam.

The truth is that bigger instances, or instances holding bigger communities hold a larger slice of the pie, and unless you forcefully remove people from the instances, or remove the instances from the people, they cannot be replaced. You can't start your own instance and communities to replace the big ones at the top. Your new one will occupy less of the pie.

Looking at the communities tab can give you an idea of which ones they are

​

The instances which hold these positions hold control, and they can't be as easily replaced or bypassed. So "Just move to another server" doesn't work unless you're encouraging ban evasion, which most people are not. They can make a real impact on your experience in the Fediverse, and trying to deny that fact is disingenuous and misinformation, because they absolutely do hold control here. They are the ones who's actions make a difference, especially because they cannot be replaced easily.