Purpose / background: Convolutional neural network (CNN), a type of deep learning technique, is an emerging radiomics research methodology that is capable of automatically extracting and learning the features of an image with only minimal preprocessing. It is yet unclear whether the radiomic phenotypes of brain metastases (BM) are related to the prognosis of radiation therapy. The aim of this study is to assess whether a CNN-based radiomics model can predict the early response of BM to radiosurgery using the phenotypes of CT images. Materials & Methods: One hundred and ten brain metastases that were treated with stereotactic radiosurgery (SRS) were selected for analysis. Post-SRS images (within three months of surgery) of the tumors were assessed according to the Response Evaluation Criteria in Solid Tumor (version 1.1) and labeled as either responder (complete or partial response) or Non-responder (stable or progressive disease). Data-sets were created by matching an axial planning CT image containing the center of the tumor with the tumor’s label. The 110 data-sets were randomly assigned to the training, validation, or evaluation group. Random assignments were repeated to create 50 independent data-set combinations, which were in turn classified into five groups. Each group contained 10 different data-set combinations that have the same evaluation group data-sets. The experiment was proceeded as follows for each data-set combination. First, the constructed CNN was trained with the images of the training group and matched labels. Validation data-sets were used to choose the learning model with the best performance. Evaluation images were presented to the chosen model for it to classify the images into responder and non-responder. Then we compared the classification output to the matched labels. The CNN used in this study was constructed with TensorFlow, a machine learning framework. Results: Of 110 tumors, 57 were classified as responders, and 53 were classified as non-responders. The prediction area under the ROC curve (AUC) of each CNN model for the 50 data-set combinations ranged from 0.602 [95% CI, 36.5%-83.9%] to 0.826 [95% CI, 64.3%-100%]. The AUC of the ensemble models, which take an average of the prediction results of 10 individual CNN models within the same group, ranged from 0.761 [95% CI, 55.2%-97.1%] to 0.856 [95% CI, 68.2%-100%], showing superior performance over individual CNN models. Conclusions : CNN-based ensemble radiomics model that learns SRS planning CT images for BM and their responses predicted the SRS responses of non-learned BM images with high accuracy. This study presents the possibility of using CNN models to predict radiation therapy prognoses with small-scale data.