How long are you letting males go before you are 'sure they are males'? Have the pollen sacks flopped down on their stalks yet? HAve you ever actually found a hermed female, or are you assuming there is one because of the persence of seeds?

If you grow with seeds pollinated by a herm you will get almost all females but they are likely to become herms under the same environmental conditions.

Sponge down all the surfaces in your grow room with a big soapy sponge to get rid of pollen.