Actually thinking back I have been turned on by some girls’ perfumes

I think you just answered your own question.

It's not the scent itself that made you horny but that it was associated with someone you found attractive. From that point on, they are inexorably linked and any slight whiff of the perfume will conjure that person up in the deepest olfactory centers of the brain. If your first introduction to that scent was on someone you found repulsive you'd likely react negatively to it.

People really jerk off to the scent of a person

if that wouldn't be a thing, people could hardly sell their used panties, right?

i'm telling on myself, but I love sniffing undies. it's the next best thing to being in their crotch.

I wouldn't buy a random person's underwear, but i've smelled my (now ex's) undies, or taken a whif of a hookups...