I think it's oil that absorbs and holds onto the smell. Some people are naturally oilier than others. If your clothes are fresh and clean and you have recently showered the smell doesn't stick around, that's for sure. You don't have to be dirty for it to stick, the natural skin oil people have can hold onto it.

Items of clothing like jackets that aren't washed everytime you wear them hold onto the smell more because of this.