IDK, I have some "socially progressive but not deep into politics"-type friends who said they liked it back when it initially released. I think a lot of people just got engrossed by "pretty animations" and collective hype and just take the "the protagonist 'learns to be better' and 'forms a bond' with the slave girl" plot point at face value. Like, a lot of people don't realize the gross framing at work because the story appears to be condemning the bad behavior (even if it's in a dishonest manner). The backlash to the slavery fetish genre wasn't as strong at that point so I guess the incel vibes were less obvious back then.

That being said, a lot of the people who are still into the fanbase at this point are probably creeps who are into the slavery.