Conventionally, there needs two equations to solve the traditional optimal tracking problem, a Riccati equation solved in a feed-backward way and a non-causal difference equation solved in a feed-forward way. In this paper, an alternative augmented system is introduced to show that in the time-varying case, the tracking problem can be resolved in a similar way as the regulation problem since they are equivalent. In this study, we employ a reinforcement learning method to develop an off-policy policy iteration algorithm that solves the linear optimal tracking problem for time-varying systems with unknown system dynamics. The simulation example provided in this paper demonstrates the effectiveness of the proposed approach.