I wouldn't worry about it too much. As long as they aren't eating at your leaves or savaging your soil, I wouldn't fret. I see a few little aphid like gray bugs fly around my plant, they don't bother it.

I'm curious, what are you growing in?