Mold in on first harvest

This is my first harvest and my plants are in the flowering stage. Recently when trimming I noticed brown mold on five different buds, so I decided to do a early harvest and cut off all of the moldy buds and start drying the buds that didn't have mold. Is this a good idea or is the entire plant lost because of a few moldy buds?

thanks for the replies.

Toss what has mold and keep rest. Just look at all buds to make sure they dont as well. Drop the humidity and put fans on to make more airflow if any more days in bloom.