If he was growing weed I don't think he would call the cops on you.

If it's an old man or even a bitter middle aged person, if it's there property, they just don't want you on it, that's all, so stay off his land because it is none of your business.