Developing effective helmet detection is very useful in construction sites. To explore the applicability of YOLOv4 in helmet testing, a new helmet detection system PFG-YOLO based on the YOLv4 network is proposed. The up-sampling using interpolation method is replaced by PixelShuffle which solves the losing information on fusing low-resolution with high-resolution feature maps. The ghost module is introduced to reduce the FLOPs and parameters of the network. The fire module is introduced to reduce the model size. When comparing the performance of the two models, the PFG-YOLO we proposed, not only increases the accuracy but also decreases the complexity of the network model. The mAP of class helmet and class no-helmet increased about 5%, both parameters and FLOPs at the corresponding layers were reduced by about a half, and the model size reduced about 20%.