I usually go by the rule that if the leaf plucks right away from it's base with just the lightest tug, then it's okay to let it go. If you have to pull so hard that the branch pulls along, then leave it be.