What an exceptionally well-observed point. If you think of the universe as a matrix with individual instances being co-ordinates in time and space, then how indeed would you go about trying to pinpoint an instance, even if you had a technology that allowed you to travel to anywhere in time and space.
Is this the nail in the coffin for functioning time travel?
The only thing I could envisage that would allow you to time travel accurately given this point is a supremely powerfully computer that could use information about the universe and the things we perceive around us now to extrapolate exactly what physically events had taken place throughout the history of the universe, even down to the quantum level.
A macro-example of how this would work is to say, if I am sat at my laptop writing this statement at exactly this time and place, a set chain of events must have conspired to bring that situation about: I have a hangover so I'm here on my own rather than out in Hammersmith where I am supposed, and for me to have been getting drunk with exactly the people I was with at exactly the venue I was last night, certain things must have occurred. A super-powerful computer could extrapolated from this information and work out what happened in the past