I take some off during veg, no problems really.

Why would you guys say it's better to take them off during flower than veg? Isn't the plant more sensitive at that time to becoming a hermie and other problems related with stressing them?