He may be jealous of your weed.

I've known people that when they get jealous they just refuse to participate....it is strange.