I get what you're saying, and it's probably true that our hesitation towards aggression and brutality comes into play, but still think this is only part of the story.

Sure, if pressed to the limit/a life-death situation, a weakling like me can do serious damage to just about anyone, but I cannot physically rip someone's face off. And neither can you, and neither can a bodybuilder/strongman. Chimps are built different than us, and so are most animals.