Crack detection is of great importance for road maintenance. It is also a critical task for insuring traffic safety and travel safety. It is still very challenging to establish a unified and robust framework to perform accurate crack extraction from images with complexity of the background, various morphological differences, the low contrast with surrounding pavements and possible shadows with similar intensity. In this paper, an improved semantic segmentation model with reconstruction branch is proposed for crack detection, and applied in smart safety road maintenance. Based on normal segmentation network, a deep convolutional encoder-decoder network is built to learn the image reconstruction mapping. This reconstruction guided semantic segmentation is aimed at reducing false recognition rate and improving detection accuracy by introducing reconstruction difference between crack and normal areas. The experiments demonstrated that our proposed algorithm outperforms the convolutional segmentation method on two public datasets and one our private dataset.