Depending on where you live your water will either have chlorine or chloramine. If you have chlorine yes leave the tops open to let the chlorine evaporate out, it can take up to 48 hours, the more surface area the water has the faster it will leave the water. Also bubbles or waterfalls help a lot.

If you have chloramine then it can take 6 weeks for the chlorine to leave. Chloramine is chlorine and ammonia mixed. It becomes much less volatile when mixed with ammonia thus it stays in water much longer. Water companies like it better because it stays in the water longer thus water is safer in distant pipes. So it is useless to leave your water open to air to remove chloramine. I hate this fact because I am forced to use water with chlorine and its hard to remove, so it always kills my beneficial bacteria. Brew shops have good tabs to remove it.

Go to your water districts web site to find out which type of chlorine you have.
So cal has chloramine