I always say 'now isn't a good time, but if you leave me your name and address I'll call round to your house when it's convenient to me'.

They tend to leave faster than Benny Hill on speed!