Adversarial attack in Visual Object Tracking (VOT) aims at deceiving trackers by adding imperceptible perturbations to video frames. Despite there have been numerous adversarial attack methods developed for VOT, achieving a balance among the criteria of efficiency, real-time performance, and stealthiness remains a persistent challenge. To address this issue, Blur-Shift attack is proposed to effectively fool the Siamese tracker by solely perturbing the tracking template while ensuring remarkable real-time performance and stealthiness of the attack. Concretely, a highly efficient perturbation generator is trained with the blur loss and the channel-shift loss, which can shift the predicted box to the edge of the video frame with a natural adversarial blurred pattern added to the template. Substantial experimental results on OTB100, LaSOT, and VOT2018 illustrate that our attack method superbly satisfies the three criteria mentioned above when targeting SiamRPN++. Furthermore, our method has great transferability and is able to deceive other state-of-the-art trackers such as DaSiamRPN and SiamMask without retraining.