The fan is doing more than removing excess heat. The air exchange is why you need the air. Like having a bunch of people stuffed in a car with the windows rolled up, the air gets stuffy and stagnant, open a window and everyone goes Ahhhh.

You need the air flow period