Alright, so I found out that I was over watering the plants, thats why the mushrooms were able to grow. Now that I have backed off the watering the mushrooms have gone away. :-)