I was having the same problem.

Check out what I did, no permanent changes to the original structure at all. (once you put the door back on of course, simple enough though...) If I open the windows at night, letting the cool night air in, then close them up before the temp starts to raise in the morning, I never even have to touch the A/C. On an 85 degree day, the closet didn't raise above 80 with fans on, no A/C all day.