I've heard you can pluck them off. I assume you'd just pinch them off at the base... as close as you can get to the stem.

If they keep coming back I would weigh your options as to whether these plants are worth a possible pollination of the rest of your crop?

What strain is it? Were they stressed in any way that you can recall? Overwatering, pH, overfeeding, etc....