Ya, if i were you, i would be worried about your friends telling everyone that your growing pot. its not like they would go to the police or anything, but if word got around in the smoking community in your area that your growing, and some cop catches one of these guys smoking one day, they are most likely to give your name as a person who could potentially be growing his weed not for personal reasons but for profit instead of a dealer that they actually get their weed from.
i hope that makes sense