If the worst thing Nazis did was go around sporadically punching people, they would not be the universal byword for evil.

Going around trying to exterminate people makes you as low as the Nazis. Going around punching Nazis makes you a damn sight better.