We present a novel ensemble-based method for the accurate and reliable classification of some of the most common types of brain tumors: meningioma, glioma, and pituitary tumors. We show that using the proposed method, we can re-use the same neural architecture and build diverse ensembles that surpass the performance of the original architectures and could potentially be used as a part of a Computer Aided Diagnosis system. In this paper, we demonstrate that we can treat the features extracted by the convolutional layers of the members as elements of a latent vector space and use cosine similarity as a measure to calculate their similarities. By using a custom loss function and regularizing the members based on their internal similarities, we show that we can train diverse ensemble models that can generalize well to unseen data and classify brain tumors with high accuracy, sensitivity, and precision. We experiment with multiple state-of-the-art CNN models that are frequently used in the clinical setting and show that we can surpass all of them in terms of performance, achieving over 91% accuracy and precision and over 90% sensitivity and F 1 -score.