Personally, I would be a bit worried, but I wouldn't think much about it at all, though. I'd make sure my plants are completely out of sight (if at all possible, might want to think about re-locating them for a bit) and that the smell is gone.
Then, I wouldn't give one FUCK one those pigs do. I'd wear my Wake and Bake shirt (actually, mine is Always Ready to Bake with that talking dough character on it, dough boy?) and tell them to gget out my house.