I can see this person's concern. There are a lot of people out there who would not want their young kids to stay alone in an environment where they might be exposed to illegal drug use, regardless of how you personally felt about the habit or the people involved.

Also, it is totally wrong to get the kids excited about the possibility of visiting if they haven't cleared it with the parents first. Not cool to put parents in the position of having to be the "bad guy" by offering something to the kids that the parents won't allow.