You definitely have to worry about it pollinating. That stuff sticks to your clothes, pets track it around, it flies through the air with ease - I'd keep it far, far away from your grow room but that's me

You can try snipping the balls off, but you still risk pollinating and ruining your hard work.