Law enforcement types have better things to do than look for every single person that is growing for personal use(1 or 2 plants). They are looking for those growing for sale/distribution so it looks good on the 6 o'clock news. 99% of the time when somebody gets busted with a half dozen plants or less, its because they were running their mouth, or some nosey neighbor was poking around somewhere they shouldn't have been, or a pissed off EX turned them in.
If you can keep your mouth shut and don't advertise the fact that your growing, you won't have a problem.
When in doubt, common sense applies.