I always read the joke as "there's an attractive woman" and the punchline is "I'm attracted to my friend in a way I wasn't aware of prior".

At least, that's what I'm laughing at.

I do see the validity of your point now that you point it out.