Keeping the soil too moist for too long will develop mold issues. And since mold isn't stationary, it can migrate to the buds, expecially in the presence of wind and humidity. (fans and foliar spraying in flower, for instance...)

Doing nothing about the mold risks exposure to the active spores via handling and smoking. Plus, the spores will hang-around waiting to infect your next crop. Not an optimal situation, especially for a medicinal gardener or caretaker.

Fix the issue, or risk health issues for you and for those you share with.