In the intelligent high-speed railway system, the security of deep neural networks-based high-speed train bogie fault diagnosis methods is challenged by adversarial attacks, which can mislead the model predictions with maliciously designed adversarial examples. However, existing methods do not consider the robustness against adversarial attacks. To address the aforementioned challenge, we propose a novel method called AdvSifter to perform robust fault diagnosis for the high-speed train bogie against adversarial attacks, which leverages adversarial training (AT) to guarantee the model with fundamental adversarial robustness. Besides, a defense algorithm called residual perturbation inversion (RPI) is developed to recover and remove the perturbations in adversarial examples to reduce the power of the adversarial examples. A defense module called SifterNet is designed to perform RPI to further improve the adversarial robustness of AdvSifter on the base of AT. Experimental results on a high-speed train bogie monitoring dataset demonstrate that our method outperforms state-of-the-art methods by a large margin.