I'd pick a spot, and keep them there. Switching back-n-forth from indoors to outdoors is a great way to stress the plant (uneven lighting schedule and light spectrums) and is a great way to introduce stuff like insects, spiders, roaches...into your home. The bugs hide under leaf litter, and in the drain holes under the pot.