Hi,I think that you can leave them as so long when you see that the plants are dying.
I think,that the frost dont affect the potency,maybe only the flavour of the buds.