It's partly a joke (your favourite peeve, for example) and probably a reference to the other, now archaic, meaning of pet:

fit of peevishness, offense or ill-humor at feeling slighted

I think you can accept idioms as they are or you'll be endlessly feeling like one saying or another has got your goat