As long as the pods have not opened yet, you can relax.
It takes a week or so for a male to drop pollen after it declares it's sex.
So, it's not a problem.
If you want to grow him out for pollen, just be sure to keep him as far away from the girls as you can.
Pollen is airborn and can and will travel long distance on the wind.

Aloha,
Weezard