A fan helps to make the plant stronger from the wind blowing over it. Also helps to keep the air circulating which is important to plants. They need fresh air. If you don't have a heat problem you could shut if off for periods of time each day with I would think no adverse effects.

My one question though would be why are you turning the lights off if they're in early veg when the large majority of 'experts' suggest 24/7 lighting is best?