The resonance of magnetic levitation turbomolecular pump (MLTMP) system induced by whirl mode is a crucial problem in industrial applications. The paper studies the problem that the nutation instability is easily caused by the whirl mode control of the MLTMP system at the rated speed. The nutation phase compensation method based on the phase shaper is proposed to avoid the instability of the active magnetic bearing(AMB)-rotor system caused by the phase lag introduced by the notch filter. Firstly, the complex coefficient single-variable model of AMB-rotor system is established, and the filter cross-feedback is used to control the nutation mode of the system. Then the phase shaper is designed on the basis of the series notch filter, in order to provide sufficient damping for the nutation mode and ensure the stability of the system. Finally, the simulation results using the double-frequency bode diagram method show that the proposed method has a good performance in improving the nutation stability of the system.