I hooked up my trusty old Logitech speakers to see if they still worked. They did, so I booted a game and sheepishly turned the volume way down in fear of annoying my neighbors, immediately remembering why I had these stuffed in a closet for almost 10 years.
Even if I lived in a soundproof house in the middle of nowhere I'd still be self-conscious enough to just use headphones instead 
Receiving no complaints doesn't mean much unless they are confrontational people who like complaining.
I set my speaker baseline by putting some music on, going outside and if it's louder than ambient noise I go back inside and turn it down. Once I'm comfortable it's not loud enough to bother anyone I leave the speaker setting put and use headphones on the occasion I want a more immersive experience.