Not all harm in our society is simple one-on-one "I swing my arm and strike your face" harm. We live in a complex social order where all actions taken interact with the broader world. There are a lot of behaviors that don't cause direct immediate harm to others but cause a degrading effect on society at large when enough people are doing it.