If it rub's off, it's likely either something you've sprayed on them that now rub's off, dust, or mold.

If it's from an additive or treatment, rinse (spray) with clean water after treatments.

If it's dust, you'll need to do some housework. Dust shades light from getting to the leaf and can taste like crap when smoked.

If it's mold, you'll need something to kill the mold, more fresh air, fans for circulation and lower humidity.
Are you spraying them with anything? If so, and it is mold...stop. The added moisture will accelerate mold issues. (tiz ok to spray mold treatment, if necessary) Larger pots tend to retain water longer, and are an added source of localized humidity in a stale-air enviornment.