That's what happens when we live in patriarchy, comrade. Although obviously women and other genders have it worse, men get the most insidious scares placed in us, preemptively interfering in our relationships to recreate patriarchy through us. At least now that you have noticed it you can work on it, overcome it, and try to stop it from happening to others.