Think of all the productivity that could be done in those 5 minutes. I make a habit of never asking for permission i just do it, my colleagues know me for vanishing mid conversation and by the time they realise I'm already halfway home.