For distracted driving behavior in real driving scenarios there are problems such as complex detection backgrounds, different target scales, and weak anti-interference ability. In this paper, an improved YOLOv5 model is proposed. First, the PAN in the neck network layer of the model is modified to a Bidirectional Feature Pyramid Network (BiFPN), which improves the ability of feature fusion of the model. Next, three Convolutional Block Attention Modules (CBAMs) are introduced into the detection layer to enhance the extraction of valid information from the model, and the traditional NMS is replaced with DIoU-NMS, which improves the correspondence between the generated frames and the target frames, reduces the missed detection rate of the model, and further improves the detection performance of the model. Then, a dataset containing 14 types of distracted driving was built and augmented using data augmentation to improve the generalization capability of the model. Finally, experimental results show that the improved model achieves precision and mean average precision of 91.6% and 89.2%, respectively, which are 7.5% and 1.6% higher than the original model. This demonstrates that the improved model can effectively improve the detection performance of distracted driving behavior.