The safety status of railway track components is of paramount importance for ensuring railway transportation security. UAV imagery based fastener status monitoring is an effective alternative to manual inspection. YOLOv7 has demonstrated outstanding performance in the field of object detection and has been widely used for detecting abnormal conditions in railway scenarios. However, the deep layers of the network have led to a large number of parameters, resulting in a slowdown in inference speed. Jierun Chen et al. proposed a new partial convolution (PConv) to achieve a faster network structure, which efficiently extracts features by reducing redundant computations and memory access simultaneously. Taking inspiration from this, we propose a new single-stage detector called PB-YOLO for faster inspection. Firstly, we create a new efficient aggregation-based network with PConv called CPC to replace the original ELAN structure. Secondly, in the feature fusion network, we design a repeatable bidirectional cross-scale path aggregation module, which dynamically fuses features and facilitates information exchange, enabling the network to handle the features having different scales and semantic levels for improving detection accuracy. Finally, experiments conducted on the customized track dataset UAV2022 show. PB-YOLO achieves excellent performance with an average precision (mAP) of 96.7%, and reduces the parameter count by 8% and GFLOPS by 20%. The ablation experiment verified the effectiveness of each module of PB-YOLO.