Like psteve said, find new friends. If you don't like hanging around them, then stop hanging around them. If they ask why you aren't around anymore, just tell them why, but in a nice way.

Keep in mind they may not take in a good way, even though you are being nice.

That's what I would do.