Confidence is key. Stare him in the eye, dont shake, dont make agressive movements, but stand your ground, and tell him how you don't want him doing that anymore.

Or you could just get some tough buddies to come along with you.