To improve the accuracy of car paint defect detection, this paper used a deep learning method to realize car paint defect detection based on small dataset. Aiming at detecting car paint defects and improving the accuracy of the detection algorithm, this paper proposed a Mobile-Transformer algorithm for automatic detection of car paint defects by replacing partial Convolutional Neural Networks (CNN) with the Transformer architecture (Transformer), improving the feature layer of the network, and optimizing the parameters. The Mobile Transformer algorithm combined the benefits of CNN and the Transformer architecture, learned global features of car paint defect images, and built a lightweight, low-latency mobile vision task network algorithm. In the training process, since the car paint defects dataset was a small dataset, it was necessary to introduce the transfer learning method to improve accuracy. The Mobile Transformer algorithm was used to extract the image features to complete the automatic detection of car paint defects more efficiently. Experiments showed that the Mobile-Transformer algorithm realizes the lightweight of the model, and the accuracy is as high as 99.5%, which is 9% more accurate than the Vision Transformer algorithm, and 5% more accurate than the MobileNet V2 algorithm.