"Funnily enough, everyone you insight check seems to be avoiding you and murmuring to one another behind your back, some times pointing at you when they think you wouldn't notice."

Because you're acting like a loon and people don't want anything to do with you.

...or maybe because the whole town is a big cult? Who knows!