Might you be using a mushroom compost? (usually for outdoors, and buffered to a lower ph.) If used in containers, it can be a bitch to control ph.

Pull 'em and toss 'em. No big deal.