Is that soil all sandy like that? If so, it's likely starving your roots of oxygen. Wet sand takes a while to dry out, not to mention it's not a good solo medium to be using.

I would find a more appropriate potting mix, one that has perlite mixed in for drainage and aeration.

Give us any information on that soil you have, a picture of the ingredients or a name or something so we can determine if it's the culprit. My guess is that it is.