Well, from what you described it doesn't sound at all like spider mites. Although I've heard that spider mites get on your soil, I doubt you could really see them down there.

I'm just fighting through a bout of spider mites now. Here's a pic of what damage they do to the plant leaves.

As you can see, they make tiny holes in the leaf tissues and suck out all the filling, leaving a tiny spot. Spider mites eat the leaves SAP. They will starve in the dirt. Not only hard to see because they are so small, walking across your soil is like hiking sand dunes for these tiny vampires.

I think you have something else!

Good luck