An old electric-shaver cleaning brush works great. 1 side has the stiff-bristles & the other side has like a mini-scraper on it.
I use it after each grind...that way there's no build-up.