I've never had a problem with Neem.
In fact my plants seem to love it.
I use it up to 3 weeks before harvest.
However I do not spray all the time and I use a very diluted solution 1/8 tsp ( or less ) per quart.
If your using Neem in hot dry conditions or over using you will run into trouble...