A robust algorithm that is capable of predicting nutritional deficiency in rice plantations is vital for enhancing the current status of associated agriculture. The research presented in this paper aims to obtain an optimal algorithm that is capable of accurately predicting Nitrogen, Calcium, and Potassium deficiencies in a self-collected dataset. The paper further leverages four famous deep learning architectures namely, ResNets, DenseNets, MobileNets, VGG-16, and SqueezeNet. All these approaches are thoroughly compared on multiple performance metrics and this paper also provides information concerning the associated temporal characteristics. The paper also assesses the use of finetuning pretrained models through rigorous experiments. The novel dataset is rigorously tested and the proposed architectures are also subjected to thorough hyperparameter tuning. A stratified train test split is leveraged to maintain the unbiased nature of the study and an in-depth empirical breakdown is presented.