In recent years, the real-time video monitoring of helmet wearing based on deep learning has attracted extensive attention. However, due to the high computational complexity of deep learning model, it leads to serious consumption of computer memory and poor real-time detection. In this work, a method of combing MOG2 and YOLOv4 was developed to improve the detection efficiency of helmet wearing in video stream. MOG2 is helpful in removing the interference of static redundant frames and save computing resources. YOLOv4 was used to identify whether workers wear safety helmets. The experimental results have shown that the proposed method has good practicability in moving target detection.