Solid buds- yup, if neem gets in the buds they'll be NASTY.
So from the time you first see sex, you've got about 2 weeks to deal with the problem. After that, you risk smoking fungicide.
Remember average flower time is 8-9 weeks. Sex shows during the second week for males and a few days later for females.