yes the males would pollinate your females. maybe you could try to sex all the wild males and kill em?