Neem is nontoxic to humans but tastes ass nasty. Shit. I'm sorry to hear about that. I hope you didn't get a bunch on the buds themselves.
In the future, the correct anti-fungal treatment is Serenade and peroxide. Serenade can go on the low leaves and peroxide in the res. In soil grows, a bacillus soil drench is also an option.