Only take over what you are happy smoking, and only go if you feel they are your friends.
It's a difficult situation really, because they could like you for more than your buds and just genially enjoy having a smoke with you.
Although if you think they are only using you for weed, they probably are. Which is no reason to chill with them, even if they are fit.
Surround yourself with people who are genuine.