It is 99% certain that your tap water is not the issue. It almost never is, and most cities now use Chloramine instead of Chlorine, and the only way to get it out of the water is by charcoal-filtering. You don't have to leave tap water sitting out.

Yes, flush them very gently with plain tap water next time they are thirsty and due for a watering. You can use a squeeze-bottle like a ketchup dispenser if you want to avoid messing with the roots too much. Try to get at least twice the volume of the pot to run out the holes in the bottom, or up to 3x. Then give the light foliar feed and let them soil dry back out slightly.