Also be wary of the container temperature. If it is too hot out and you leave it in direct sun, your medium will heat up as well and possibly bake and kill roots.

As others said, bugs are a huge concern. Anything you may bring into your house on a plant from outside could colonize and damage future grows as well.

That all being said, it's been done before. I have done it (though I'm not sure it benefited the plants that much) with no problems. Just make sure to keep em watered, and inspect the hell out of em for little beasties.