Bummer. Doubtful that perlite is good to smoke.

Maybe wait till it is harvested and dried?
Maybe a hair drier and a small paint brush? (not the sponge type)
Maybe a light spray with ph'd water as you are gently brushing chunks away? (trichomes are not water soluable, so they should be fine...)

Perhaps a can of Red Bull, and a pair of tweaker-tongs? (tweezers)