Cracks are one of the most common and significant types of road surface damage, posing a threat to the safety of pedestrians and vehicles. If left untreated, cracks can lead to severe consequences such as road and bridge collapse. Therefore, it is essential to develop an efficient road crack detection method. Traditional crack identification methods have the problem of being largely affected by the environment and having low recognition accuracy. In this paper, we propose a road crack detection model based on an improved You Only Look Once version 5 (YOLOv5) model that addresses the limitations of existing state-of-the-art crack detection methods in terms of accuracy and detection speed. First, we replace the intersection over union (IoU) loss function with the SCYLLA-IoU (SIoU) loss function for better accuracy. Second, to enhance detection performance, we replace the feature pyramid network (FPN) with a bi-directional feature pyramid network (BiFPN). Finally, to better extract spatial feature information of different sizes, we modify the original Spatial Pyramid Pooling-Fast (SPPF) module of YOLOv5 by using Spatial Pyramid Pooling Cross-Stage Partial Connections (SPPCSPC). We evaluated our YOLOv5s-BiFPN-SPPCSPC-SIoU (YOLOv5s-BSS) method on the dataset from the IEEE 2020 Global Road Damage Detection Challenge (GRDDC) and achieved promising results on road damage datasets from China, Japan, and the United States. The mAP@0.5 of different cracks in three datasets reached 84.9%, 54.6%, and 71%. Our method outperforms related methods, with an increase of 0.7%, 0.7%, and 2.8% over YOLOv5s.