Might you be using a mushroom compost? (usually for outdoors, and buffered to a lower ph.) If used in containers, it can be a bitch to control ph.

Pull 'em and toss 'em. No big deal.
Rusty Trichome Reviewed by Rusty Trichome on . WTF!!!?? Ummmm... So there are 3 baby mushrooms growing in 2 of my pots.... WTF!? What the hell do i do?? Just pull em out? how did this happen? Rating: 5