I usually reply with this, " am I a stoner? No, but I do smoke bud from time to time."

If I say yes, they will think it is because I want to be cool or whatever.