I personally would like to agree with you because I prefer universalism, and I think the alternative, utilitarianism, leads to extremely bad outcomes in practice, despite seeming like a pragmatic option in theory.

However, is it perhaps also true that another one of the reasons we cannot have nice things is because we cannot practically do away with those who would themselves never hesitate to increase the net suffering of humanity if it benefited themselves?