Well, I myself suffer from mild allergies, and cannabis use has in now way harmed me. Perhaps the smoking process has taken down my immune system slightly, but other than that, I'm fine.

One of the many differences between cannabis and tobacco is that cannabis widens the alveoli in the lungs, whereas tobacco shrinks the alveoli. This is why it helps some people with allergies (I would say it helps, but I don't really notice a difference with pollen)

I guess what I'm trying to say is, unless you have STRONG allergic reactions against pollen, you should be pretty much okay. I know people with asthma who smoke fine, though they usually have to take smaller hits. Just don't psyche yourself out! It's important to be happy and calm, you'll be okay.