Once, while my grandma was out of town, i was sitting out on her balcony smoking a j.. only to later see her neighbores in the yard next door, downwind, sitting out staring at me. It was rather awkward, but as far as i know they didnt do anything about it.
Just be careful about when and where you smoke, put a little thought into it and you should be good.