The paper proposed a novel Joint Detection and Segmentation road scene understanding deep Network, named JDSNet, achieving real-time inference with high-accuracy result. Overall, we accomplish this by four components: 1) Based on encoder-decoder network and global pool module, Shared Feature Module is designed to obtain richer multi-scale feature. 2) Adaption Feature Selection Module is developed to learn task-specific features, it’s considered as feature selector from Shared Feature Module. 3) Task Cross Module is proposed to enhance subtask information exchange. 4) Dynamic Weight Average is introduced to average multi-task loss weight over time by considering the rate of change of loss for subtask, to find good balance between tasks. Finally, we evaluate JDSNet on Cityscapes road datasets and produce competitive results compared with the state-of-the-art methods. Specifically, JDSNet achieves 75.3% mIoU and 56.4% mAP on Cityscapes dataset, with speed of 24.2FPS for 1024×512 high-resolution image.