That's because atmospheric gases dissolve into water that has been left out (air-equilibrated) and the CO2 goes into an equilibrium with the carbonate ion and carbonic acid and acts to raise the pH to near neutral. That is NOT a bad thing. Read the chemistry link in my sig for some background.
You don't need to let tap water sit out. Chloramine does NOT evaporate- it must be removed with activated charcoal or a RO unit. In non-organic soil grows, it's unlikely ever to be a problem for you.