With increasing inroads of IoT in agriculture, there is immense scope to re-think how pervasive monitoring of crop can play a pivotal role in management of crop stress due to pests and diseases. There is a need to evolve integrated stress assessment strategies that start from forecasting likely stress to managing it at an early stage of infestation, which requires precise insights from granular data. For a given region of interest, spatio-temporal monitoring of ambient conditions with a dense set of micro-climate points and in-field collection of stress events through mobile sensing creates a pervasive farm-IoT infrastructure that generates rich contextual data. Along with nuances for crops, associated pests, and agro-climatic zones for the region of interest, this data can generate insights for effective integrated stress assessment. With one of India's major fiber crops, cotton, we present our work in the state of Telangana and Andhra Pradesh that was monitored intensively for Kharif season of 2020. For ambient parameters along with prominent pest and disease incidents gathered from the ground, we propose prediction models that significantly enhance the quality of forecast. To ensure best utilisation of the data in developing and validating the models while helping early action on the ground, we propose deep-learning based recognition and labelling of pest and disease related stress in images associated with each data-point. The approach creates a scalable virtuous cycle where insights generated from a variety of data forms work together to achieve effective integrated management of crop stress to help farmers.