The occurrence of pavement surface distress seriously affects the driving safety, especially on the driverless road. It is urgent to find a fast, automatic and intelligent pavement distress detection method. Aimed at common distresses such as cracks and potholes on driverless roads, an improved semantic segmentation network based on U-Net network was proposed, which can effectively realize pavement distress detection. Firstly, the dataset was collected, and after pre-processing, it was divided into training set, validation set and testing set according to the ratio of 8:1:1. The data enhancement method was used to polish the dataset. Secondly, the U-Net network structure was improved to adapt to the distress's multi-scale characteristics, and the loss function was adjusted to improve the proposed model's accuracy, which can avoid misdetection and missing detection. Finally, the experiment results show that the network has a good accuracy rate (more than 85.0%), which promotes the rapid and accurate detection of pavement surface distress, and ensures the driving safety.