It's the same as every other person who does not or cannot vote. But I'm somehow morally worse?

Because to them you being on the left means your vote should default to Democrats, so voting (or not voting) otherwise is an active choice compared to the passive “default” of voting Dem.