Hyperspectral object tracking (HOT) has received increasing attention due to the property of hyperspectral Image (HSI) underlying material information of background and foreground, and it shows the potential to outperform conventional RGB-based tracking methods, especially in complex scenarios. However, it is difficult to train a powerful hyperspectral object tracker due to the lack of large-scale HSI-based datasets. To address this problem, we develop visual prompt hyperspectral object tracking (VP-HOT). In order to leverage the knowledge of the fundamental RGB model, our methodology involves the freezing of pre-trained model parameters and the incorporation of hyperspectral images (HSIs) as visual prompts, with only 1.7 million trainable parameters. This adaptation improves the suitability of the model for HOT tasks. Extensive experiments demonstrate the potential of visual prompt learning in HOT, and VP-HOT achieves outstanding performance across diverse downstream tasks, spanning VIS (16 bands), RedNIR (15 bands), and NIR (25 bands).