You'll get mixed replies with this question.
I've smoked leaves before, from both male and female plants that were not flowering. It tasted like absolute garbage and made me gag, it burned my throat too. It did give me a little buzz though but only lasted about half an hour then I just felt tired. It also depends on which leaves you smoke, the growth shoots will be a slightly better smoke than the fan leaves (still shit though). I think it may depend on the strain too. So it's really not worth smoking the leaves, unless it's those small leaves which grow between the female flower buds and are covered in resin, they'll get you high.