Whats the best way to remove dust from the outside of my carbon filter?

I bought a carbon filter system a few weeks back and I stupidly thought that the white dust sleeve on the outside of the filter, was to protect it in transit and I removed it. I have now bought a replacement sleeve, but before I fit it, I want to remove the small amount of dust on the outside as I think it is affecting the filters operation. Would it be OK to a vacuum the outside of the filter? or will this damage the filter/suck out carbon?

If this is a bad idea, does anyone have any better suggestions?