Rice diseases pose a great threat to abundant yield, stable harvest, and high-quality production of rice in China. Among them, the four diseases causing the most significant yield loss are sheath blight, rice blast, false smut, and ear rot. Accelerating the diagnosis and accurate identification of rice diseases is very important for the future development of rice production. To enhance the diagnostic accuracy of traditional CNNs in small-sample rice disease image sets, this paper proposes an ACNN-TL model based on CNN structure combined with Atrous convolution and transfer learning. Compared with standard convolution, Atrous convolution can increase the size of the receptive field in feature extraction and enrich the extracted feature details. Transfer learning can use knowledge in the original model to enhance capability in the target task. The experimental results show that the accuracy of ResNet-34, VGG-16, and AlexNet combining Atrous convolution and Transfer learning is 98.4%, 97.9%, and 95.9%, which increased 8.7%, 9.4%, and 16.7%, respectively, compared to with the original model. It can be seen that the combination of Atrous convolution and Transfer learning can effectively improve the diagnostic accuracy of CNN for small sample rice diseases, and the best model is Atrous ResNet-34 with Transfer learning.