No, leaving them in the bud actually lets them ripen further. I let my seeds ripen in the buds, and then take them out when the buds are dry and they just fall away.