what you're missing is that there is no "past". Time is an illusion. And secondly, to get a somewhat accurate idea of how time travel would actually work watch the most recent Harry Potter movie (I know, I know....) but all jokes aside, the way it's portrayed in the movie is actually theoretically correct.