In order to effectively detect the high-altitude falling behavior in the surveillance video, an automatic detection method of high-altitude falling based on the improved frame difference method(IFD) is proposed. Firstly, the video data are processing for debounce, and the optimal number of interval frames is determined with the proposed method. Then, the motion prospect is obtained by using the optimal number of interval frames. Finally, a fall judgment criterion is proposed to determine whether the motion prospect is a high-altitude fall or not. The experimental results show that the proposed method can process surveillance video efficiently compared with frame difference method and three frame method. In this paper, the problem of low efficiency of other methods in detecting high-altitude falling objects is solved with IFD.