Just kick him out next time. When he starts bitching you can just tell him, "the door's over there." You'll get the point across. Be honest to this guy and tell him you don't want him around you because of how disrespectful he is. Distance yourself from these people. You don't need to escalate it to fighting over one minor thing like this, just keep him away, that way it won't ever escalate to that.