Movement interpretation from the motor imagery (MI) signal of electroencephalogram (EEG) is an emerging field that can potentially upgrade the quality of life of individuals with motor impairments. In this manuscript, an explainable deep learning (x-DL) approach has been proposed to identify four class MI form EEG signals. The acquired EEG signals are preprocessed and then converted to time-frequency (TF) spectra for extracting the frequency content of the MI signals. The obtained TF images are then fed to a deep learning (DL) framework employing residual convolutional neural network for proper classification. The classification score, obtained from the network, has been propagated through the layers of the DL framework up to the input layer for obtaining layer wise relevance maps. These relevance maps signify the explainability of the proposed network, in terms of visual results. Comparative analysis reveals the effectiveness of the proposed x-DL module using a benchmark dataset. The obtained results and the utilization of signals from only two EEG channels demonstrate the possibility of employing the suggested strategy for developing a cost-effective brain-computer interfacing architecture applicable in real-time healthcare applications.