Handwritten character recognition demands great importance in the field of bank cheque processing, tax returns, etc. Deep learning techniques for recognition of handwritten characters have surpassed the traditional techniques involving handcrafted feature extraction. Although it achieves around 0.95 recognition rate, some misclassification does exist. This is because the classifier function at the output layer performs classification that does not consider the parameter adjustments using the low and mid-level features. This paper proposes an efficient method of recognition of handwritten Malayalam characters, including compound characters and signs, using deep layered graph neural network, Residual network enhanced by multi-scaled features from lower and middle layers.