I like to wait. The extra leaves form a protective wrapping around the bud and help keep it intact. I don't pull those leaves off until I'm ready to smoke. It's like unwrapping a christmas present.

Also, while the plant is drying, it will continue to pull nutrients out of the fan leaves as it is still ripening.