The quadrotor flies for a long time so that its actuator is prone to have faults and it will be affected by the external disturbance. In order to obtain better flight performance, the author of this paper conducts a fault tolerant control design based on sliding mode observer to solve the problem of quadrotor’s actuator fault under external disturbance, the results of which is of great significance for the quadrotor’s safe flight. First, the model of quadrotor with the effects of disturbances and faults is built. Then, a kind of sliding mode observer is designed not only to reconstruct the fault information, but also to estimate the external disturbances in real time. Finally, based on the obtained observation signals, backstepping algorithm is used to design attitude controllers for the inner ring and the position controller for the outer ring to tolerate faults and compensate for disturbances, so that the quadrotor can track the desired trajectory stably. Simulation results prove the effectiveness of the proposed scheme.