Only you can decide whats best for you. As long as you don't become anti-pot if you quit and support those still fighting for it to be legalized (like it should be) then do whatever is best for you. And if these ppl are gonna spread shit about you, they weren't your friends in the first place and you don't need them.
If i'm mega stressed smoking fucks with me sometimes, but usually it just clears my head and lets me think logically instead of emotionally.