Rupture of carotid plaque is clinically important to identify patients at risk for ischemic stroke. Recently, supervised deep learning methods have been successfully applied to classify carotid plaques, but their performance depends on the number of labeled images. However, the number of carotid plaque labeling images in clinical practice is small. To alleviate the problem of few labeled images, we propose a self-supervised carotid plaque classification network algorithm named SSCPC-Net. First, we design an image block shuffling strategy to predict shuffled and ordered images based on the spatial structure of carotid plaques, which aims to pre-train a CNN to obtain the representation of plaque features. Next, the pretrained model is transferred as parameter initialization into the carotid plaque classification network to improve the performance of the CNN when the number of labeled training images is small. It can be seen from the results that SSCPC-Net can effectively improve the classification accuracy of basic ResNet under different proportions of labeled training images. Furthermore, the results show that SSCPC-Net can effectively alleviate the problem of insufficient labeling of training images, effectively improve the plaque classification accuracy, which is helpful for clinical carotid plaque identification and stroke warning.