Male pollen has been known to travel over 30miles to pollenate a female plant. However its only by strong winds and bad luck that that would happen.
Its really up to you dude. If the hairs have started changin color you can harvest it and see what happens. But if its been pollenated its not going to produce any more buds...
