You're probably experiencing what happens when a plant gets stressed. Did you buy feminised seeds? If so, they have a strong tendency to hermie if conditions provided are less than optimal.

From what you described, I'd say it was a stress issue not a pollen issue. Your yields shouldn't suffer greatly either. You'll just have seeds to pick out once harvested.