Moving object detection is one of the important applications of computer vision. How to accurately realize the separation between foreground and background is the research focus. In this paper, aiming at the problem of poor anti-interference ability of traditional codebook algorithm in complex background and the cavity-prone problem of traditional frame difference method, a pedestrian detection algorithm integrating the optimized five-frame difference method and the optimized update rate codebook is proposed. The algorithm designs a kind of double code word structure, in the process of codebook background modeling and testing process to code the code word in the book to join the update mechanism, and based on the amount of foreground points to automatically change background update rate. This design also implements an improved algorithm of double code word book YUV color space. It is used to realize the function of realtime updating background. Then, Canny edge detection operator is added to the traditional five-frame difference method to remove the noise. At the same time, the extracted rectangular area is used for filling operation and morphological processing to obtain the corresponding foreground image. Finally, the foreground image of codebook model and the foreground image of frame difference method are calculated with “and”, “different or” and “or” of pixels to obtain the precise region of moving target. The experimental results show that this algorithm combines the advantages of the two methods and abandons the disadvantages of each method in the detection prospect. It can accurately detect moving targets and has good robustness. The effectiveness of this algorithm is also verified by experiments.