I don't know where this idea that slapping someone being permissable because they say the magic words of offering a duel comes from.

I agree that it's absolutely ridiculous in a modern context where dueling has died as an institution, but historically for one to refuse a duel from a person of equal or greater social standing than them would ruin one's reputation.