I'll start things off.
This is old shit, but if you remember when Gamergate was at its height, a lot of them complained about "walking simulators," games like Gone Home, Dear Esther, etc. with very little in the way of typical gameplay mechanics like challenges that have to be overcome through skill or failure states. Gamergate dipshits seized on a white-hot, psychopathic hatred of these games, spinning the lack of skill required into bizarre conspiracy theories about game journalists promoting these as a plot by non-gamers to pave the way for the infiltration of gaming by "anti-gamers." Also because a lot of these games are about minorities, who of course GG assholes considered by default to not be "real gamers."
The thing is, I don't like walking simulators either. I've only played a few, but the only one I even kind of enjoyed was The Beginner's Guide (and even then, I don't think I would've missed out on much if I'd watched a longplay instead). The medium is the message, as the old saying goes, and the ability to engage through interaction with the mechanics is what sets games apart from other media. Walking simulators (and visual novels, but that's a different gripe) don't take advantage of this in a way that gets me invested. To me, a walking simulator feels like the equivalent of a movie that consists solely of a guy sitting in a chair and reading a story out loud.
The difference between me and a GG dipshit, of course, is that my dislike of the genre doesn't hinge on ridiculous conspiracy theories or hatred of minorities, and also that rather than wage some crusade to kick walking sims out of the gaming club, I just don't play them. In any case, though, the association is strong enough that it's something I tend to avoid bringing up.
That's definitely the risk involved in making walking simulators, you might make a bad one where interactivity is not an important part of the experience. However, I would argue that what sets good walking simulators apart from bad ones is that they actually do have meaningful interactivity. Gone Home is a good walking sim, because the experience of freely rifling through the belongings of your family is important to the overall story. It's not like it's a linear story, you piece together the lives and personalities and experiences of the characters at your own pace, partly through your own inferences based on the things you find lying around. I'd contrast this with Dear Esther, which is a bad walking sim, because the interactivity changes nothing about the experience. The story is delivered nonlinearly, but not in any coherent way based on your own decisions. IIRC the way the story is delivered to you literally by RNG, making your 'decisions' amount to nothing more than what drab environments you happen to stumble across while being fed random disembodied voicelines.
Your experience playing Gone Home and my experience playing Gone Home would probably be meaningfully different in one way or another. Your experience playing Dear Esther and my experience playing Dear Esther, on the other hand, would not be. That's the fundamental difference between the games, in my opinion.