yea sounds like you need more rubber-to-glass contact.

a larger gromet won't help because it's probably a defective downstem (mishapen) also, push on you downstem if you can, that'll help a LOT.

Also, shift around the gromet on the slider, you may find a magical sweet spot.