Convolutional neural network (CNN) has been applied to glioma classification, but there is no research result of CNN compression in this field. Lightweight CNN after compression provides the possibilities of deploying into a field programmable gate array (FPGA) or a memory limited hardware and further integrating into a medical electronic equipments. To solve the problem that CNNs are difficult to deploy to embedded devices and simple-scale CNNs have low accuracy in glioma grading, this paper proposes a method based on Knowledge Distillation to classify gliomas, and introduces an SE block into the model for improvement, so as to improve the performance with less parameters and computation. Inception -ResNet - V2 is chosen as the teacher model and SqueezeNet is chosen as the student model. The results showed that after knowledge distillation, accuracy, precision, recall rate, and F1-Score of the student model increased 3.53%, 5.68%, 3.73% and 4.69%, respectively. GPU usage, parameter number, FLOPs, and model size dropped by 35%, 98.72%, 98.7%, 98.62% respectively compared to the teacher model.