i hear ya man. big groups piss me off too. no one seems to know what they're doing, everyone's trying to impress everyone else talking about their weed, it is sketchy as all hell.
i totally like to chill by myself, i find it safer, more relaxed, and all around more enjoyable, i don't have to carry on a conversation about something i don't even care about while other people hold onto the joint for ten straight hits. a small group of friends is totally cool. one's you get along with and you know what they're like when they're high, but i totally agree, almost always, more than a small group just means frustration.