Infrared image-based object detection is challenging due to inherent low resolution and scarce details, especially when locating small objects that cover minimal region in the image. We present a methodology for detecting small objects in infrared imagery captured by UAVs, emphasizing the use of attention mechanisms to increase detection precision. Using YOLOX as our baseline network, we integrated a modified Recursive Feature Pyramid (RFP) to enhance vehicle detection in the VisDrone-Vehicle dataset. To amplify the spatial details of these smaller objects, we incorporated an attention module and sigmoid function, bridging the connection between the primary and secondary Feature Pyramid Networks (FPNs). The sigmoid function further refines the RFP outputs by filtering out background noise. Comparative experiments demonstrate that the proposed network outperforms the baseline in detecting vehicles in infrared images.