In order to address the issues of real-time performance and the low dependency between feature channels in fabric defect detection networks, this paper proposes the ESE_YOLOv5 network based on YOLOv5. Firstly, to address the relative redundancy of the neck detection network feature channels, a relatively lightweight and efficient convolution module is adopted to ensure accuracy while reducing computation and parameter volume. Furthermore, the Efficient Squeeze-Excitation (ESE) module is introduced into the backbone to optimize the dependency of feature channels, which enhances the model's feature extraction capacity and improves detection accuracy. Experimental results show that compared to YOLOv5, the proposed ESE_YOLOv5 model reduces computation and parameter volume while improving accuracy, meeting the needs of fabric defect detection for recognizing fabric defects that have similar characteristics to the background while maintaining real-time performance.