Machine learning (ML) is a powerful tool for solving stochastic optimization problems. The aerospace and defense sectors have a number of stochastic optimization problems that would benefit from the application of ML; however, people often have difficulties interpreting solutions arrived at via ML, which undermines trust, producing an obstacle to widespread adoption in these sectors. This paper introduces the Self-Explaining Decision Architecture (SEDA) for ML-based decision-making systems capable of generating intuitive explanations for their decisions in real time. SEDA makes use of a feature extraction subsystem and a sequence interpretation subsystem to identify patterns in data followed by a decision generation subsystem that determines appropriate actions based on those patterns. Internal state information from each of these subsystems is used to generate explanations of the system’s decisions. Using this information to create explanations provides insight as to the data elements the system focused on when making decisions as well as the reasoning that was used. As a proof-of-concept, we present a first implementation of SEDA using start-of-the-art deep learning components including a combined convolutional neural network and long short-term memory network with attention mechanisms and demonstrate its use on both standard and custom datasets.