Early diagnosis of breast cancer can significantly increase the chances of appropriate treatment and survival rate. The methods doctors use to detect cancer are tedious, expensive, time-consuming, and above all, depend on the expertise of the physicians. Moreover, analysis of the image sometimes leads to false-negative reports and disagreement among the doctors. In order to improve diagnostic accuracy and act as a source of a second opinion, computer-aided diagnosis systems have shown tremendous potential. But, most of them are based on a combination of different machine learning and deep learning methods, which require not only a large number of training data but also a substantial amount of time to train. In this study, we employed a recently introduced image classification method known as Radon Cumulative Distribution Transform (R-CDT) Nearest Subspace (NS) classifier to classify breast tumors from histopathology and ultrasound images. It is a supervised learning method that is non-iterative and simple to implement in classification problems. Three different datasets are used to evaluate the performance of this classifier. The accuracies for the histopathology datasets are 93.5% and 90.9%, and for the ultrasound dataset, the accuracy is 90%. We have also used two pre-trained deep neural networks called AlexNet and GoogLeNet to classify the same datasets and compare their performance with that of our method. The comparison between these neural networks and our method shows that the R-CDT NS classifier is more computationally efficient and data-efficient for the classification of breast cancer because it does not require additional GPU and can achieve competitive test accuracy even for a small set of labeled data.