Intake should be as close to the floor as possible, exhaust as close to the ceiling as possible. Sounds like your intake and exhaust are too close together, which may result in you sucking in the hot air you've just exhausted. If you can't change your configuration and separate the intake and exhaust, you're likely to be waging a constant battle against heat. Is this a closet? If so, you could try the cabinet concept. I find cabinets have a lot to offer, foremost being the fact that you can saw and hack on them all you want and nobody cares, because it's your cabinet.

Also, the ambient temperature of the room containing your cabinet (or closet) is *extremely* important. I keep my window AC unit set on 68 degrees F, and this cool air is what comes in the intake. If the ambient temperature of your room is 85 degrees or whatever, then you won't ever get it much less than that inside whatever you grow your plants in, no matter what else you do.