Careful delineation of normal-tissue organs-at-risk is essential for brain tumor radiotherapy. However, this process is time-consuming and subject to variability. In this work, we propose a multi-modality framework that automatically segments eleven structures. Large structures used for defining the clinical target volume (CTV), such as the cerebellum, are directly segmented from T1-weighted and T2-weighted MR images. Smaller structures used in radiotherapy plan optimization are more difficult to segment, thus, a region of interest is first identified and cropped by a classification model, and then these structures are segmented from the new volume. This bi-directional framework allows for rapid model segmentation and good performance on a standardized challenge dataset when evaluated with volumetric and surface metrics.