I don't ever pull off leaves. I wait until they fall off on their own or it's clear that they will fall off soon.

If you pull off healthy leaves then the branch (and buds) that leaf fed will not grow as big as it would if you left it alone.