The practical application of crop pest detection methods has been limited by the large number of parameters and computations, and we built a lightweight crop pest detection method YOLOLite-CSG in our previous research, which basically removed this limitation. However, further analysis shows that YOLOLite-CSG still has problems that affect the performance in terms of the prior box generation method, downsampling method and bounding box regression loss function. In response to these problems, this paper proposes the prior box generation method based on label box oversampling and clustering (BS-Medians), the downsampling method based on parallel feature transformation (PSPD-Conv), and the bounding box regression loss function with size difference optimization capability (FIoU Loss). We optimize YOLOLite-CSG using the above methods to build YOLOLite-X, an optimized lightweight crop pest detection method. The experiment results show that YOLOLite-X exceeds YOLOLite-CSG in crop pest detection precision and is higher than other state-of-the-art methods. Meanwhile, YOLOLite- X has a smaller number of parameters and computations, which is more conducive to practical deployment and application.