Hmm, that's a good point.

My question now is, can we solve that problem without putting the downvote back? How?

maybe downvotes still exist but are hidden to everyone but mods, that way they can still spot egregious takes or lack of community support? So I guess, instead of a downvote, it's more of a "I find this sus" button. Like a lesser report button