Object detection in aerial images is a challenging task, primarily due to the presence of a large number of small-sized objects. In natural images, general-purpose detectors have achieved impressive performance. However, even state-of-the-art general-purpose detectors fall short of expected performance in aerial images. In this paper, we propose an improved framework to enhance the small object detection performance of YOLOv7 in aerial images. We introduce Bi-Level Routing Attention into the YOLOv7 base architecture to efficiently extract features at a finer granularity. Additionally, we employ a more suitable metric for small object detection to enhance the loss function and improve detection performance. We train and test the network on the publicly available VisDrone dataset and conduct ablation and comparative experiments for the proposed improvements. Experimental results demonstrate that the proposed improvements enhance the small object detection capability of the general-purpose detector.