You want a flapper valve. Or a spring loaded butterfly. A flapper would be best for you since your system is low pressure.

Just take a piece of cardboard or hard plastic and cut it to the size of the fan opening, then take a strip of tape and tape it to the top of the opening. Than ass a little weight, to make sure the flap goes down when the fan is off. You will have to experiment with the weight so that the flap shuts but the weight does not hinder your fan. The moving air from the fan should lift the flap, and too much weight will constrict air flow. I recommended adding some window sealing foam around where the flap seats for a better seal. This will not be air tight, it will leak a little, but for your purpose it will work, the flap will stop most of the air flow inside or outside the box.

Home depot has a dryer vent that is very similar to what I described, except it contains many little plastic flaps. I have seen them in 3, 4, and 5 inch.

Good luck
Crack Lemonade