In this paper, we first present an Plant Disease Image Dataset (PDID) for plant disease, which covers many research fields such as image classification, detection, and segmentation. And then we develop a lightweight semantic segmentation of diseases based on this dataset, which provides localization information beyond the simple classification of diseases. One practical choice to make a lightweight semantic segmentation model is to apply multi-branch architecture, which adopts ResNet18 as the backbone, achieving good speed and accuracy trade-off, since it obtains rich contextual information without losing any fine spatial information. However, we note that discrimination and generalizing power need to be strengthened due to the light structure of ResNet18. In this paper, we address this dilemma with a novel Triple Stream Segmentation Network (TSNet). We first reinforce the discrimination ability of each branch by using dilated convolution, which enlarges the receptive fields and encodes more local contextual information. Meanwhile, we carefully design a novel Feature Fusion Module to combine features efficiently, which represents the feature selection and combination. The proposed architecture achieves an appropriate balance between speed and segmentation performance on the PDID dataset. For 900×600 input, we achieve 78.3% mIoU on the PDID test dataset with a speed of 118 FPS on GTX1080Ti, which is significantly better than most approaches aiming at real-time semantic segmentation.