Denying free will based on determinism always (in my experience) ends up acting like I am some passive observer separate from my body, which behaves without my input

I don't know if we should be taking the concept of an immutable "you" as a given here - plenty of philosophers have wrestled with the apparent illusory nature of the self. That said, I would argue that free will and the self, while related concepts, aren't entirely the same thing. Plenty of behaviors (some of them complex!) happen in the absence of conscious volition. Your brain regulates your body without your conscious attention; thoughts arise unbidden. Without intending to you can find yourself daydreaming or stuck in a memory of an embarrassing situation that happened years ago. There's a definitional problem where defining free will in a way that is (a) compatible with what we know about the physical workings of the brain and (b) accounts for our lived experiences of its many limitations requires positing something not bound by the classical laws of physics that's still capable of interacting with the physical matter of the brain (wherein we get mind-body dualism) or results in something with severely limited explanatory power for why people behave the way they do.