In this paper, the active vibration damping control algorithm of helicopter rotor system with trailing edge flaps (TEF) is studied. Firstly, a control-oriented vibration response model of helicopter rotor system with trailing edge flaps is established by system identification method. On this basis, in order to achieve fast and stable vibration reduction control while maintaining robustness, a control algorithm combining finite time extended state observer and barrier function adaptive sliding mode control is designed. The numerical simulation platform verifies that compared with the traditional $H_{\infty}$ control algorithm, the proposed algorithm shows better vibration reduction effect.