Oil-tea camellia is recognized as a healthy edible oil in the world. It has the functions of lowering blood pressure, lowering blood lipids and softening blood vessels, and it has high comprehensive utilization value. However, oil-tea camellia is often invaded by various disease and pest in the process of planting, which reduces its yield, how to use intelligent information technology to identify disease and pest in oil-tea camellia is of great significance. Recent developments in deep learning-based Convolutional Neural Network (CNN) have shown prominent performance in the identification of crop pests and diseases. In order to solve the problems of low precision and weak generalization ability of traditional methods for oil-tea camellia disease and pest identification, this paper introduces an efficient CNN model based on mobile terminal that accurately identifies the common types of crop pest and diseases. The strategy of two-phase transfer learning (TL) is done during the training process. And an integration deep learning model based on the lightweight convolutional network architecture, feature extraction and fusion, and model compression is developed. The proposed model exhibits promising performance in comparison to other state-of-the-art models. Notably, the model achieves an average recognition accuracy of 99.78% for Camellia Oleifera diseases. The proposed model also shows strong generalization ability, it attains an average recognition accuracy of 99.55% on the widely utilized Plant village dataset. The model achieves recognition accuracies of 68.15% and 95.09% on the open-source IP102 dataset and the Real-world dataset respectively. The experimental results conclusively affirm the feasibility and efficacy of the proposed approach for the precise identification of Camellia Oleifera diseases.