I say U should take out all the males U see and leave the females to finish flowering.
then in like 2-4 weeks from now I would go back and harvest them.

cuz if someone planted them that close to my house. whach out.
cuz if they don't ever come around and take out there males then I say the plantS R up for grabs.

but thats my 2 cents about this.