Im not sure taking the refelctor off would help anything.. maybe check you cubic feet(LxWxH) and then look at the cfm rating on your fan.
Your fan should be able to move the volume of your closet every 3~4 mins.
Also run your lights in the coolest part of the day.