As a vulnerable large man I definitely feel this. I've had multiple situations where people just were afraid of me, like running away from my silhouette at night, and it does feel awful.

But, that's the fault of others who have abused people with their stature and "strength" in the past. Being a kind hearted giant ogre is still possible, even if at first glance people might think otherwise.

As for the rest of what you're going through, I can't entirely relate to be honest, but there's plenty of folks here who have gone through similar things so this is a great place to talk about it.