Lemme guess... the fan runs only during lights on.
At lights off, the fan shuts down. But the lights are still HOT from having been on, and the room will actually spike up to its max temp in the half hour or so after the lights go off...
If your thermometer has a probe that allows you to look at it from outside, take a peek at what the temps do at the end of 'day'. You may be surprised.

To fix, put the exhaust on a different timer that comes on at lights on but shuts off an hour later.