A few male flowers should not effect the potency in my opinion, nor would seeds. If it does minutely I dont think you will be able to tell by looking at them or smoking them.

One love
c