Ear-based identity recognition methods provide an alternative and effective means of identity recognition. Research on the identification and classification of ear can promote the development of identity recognition, particularly in criminal investigation applications where only profile facial photos of individuals can be obtained, making it impossible to perform facial recognition. In this paper, we propose a novel method for ear-based identity recognition based on transfer learning techniques to achieve improved performance even with limited training data and reduce the computational and memory resource requirements. Firstly, we transfer pre-trained weights on large-scale datasets to improve the feature extraction power of the model. Secondly, we fine-tune the pre-trained weights to make them more suitable for ear image recognition and classification tasks. Experimental results on a small-sample dataset show that our model based on transfer learning achieves an accuracy of 88.23%, which is higher than popular models such as MobileNetV2, ResNet50, and VGG16 with randomly initialized weights.