In order to improve the classification effect of potato early blight (mild, moderate, severe), the study improved the GhostNet model of the lightweight convolutional neural network, including the use of ReLU activation function to replace the sigmoid activation function of the SE attention mechanism, Dropout measure to optimize the network. At present, the most commonly used cross entropy loss function does not consider whether the data is balanced. In order to solve the problem of data imbalance, the LS-CCE (Label Smoothing Complement Cross Entropy) loss function is proposed. Through training on potato early blight, late blight, and health image data, a potato disease classification model is obtained. Its recognition accuracy rate can reach 97.17%, which is 1.046% higher than the average accuracy rate before improvement. Using transfer learning, the trained model is used to classify the degree of potato early blight, and its recognition accuracy can reach 93.27%.