You aren't kidding about using a mask and it probably won't hurt to wear gloves too.

Perlite doesn't really turn brown after waterings and feedings but it definitely gets less white and seriously unless you are planning to use it straight in a clearing with nothing but pure dark soil no one is going to notice it.