Potholes are one of the most common forms of road damage, and they can have a negative impact on vehicle conditions, comfort while driving, and road safety. For the Advanced Driver-Assistance System (ADAS) to ensure safe driving, accurate segmentation of road potholes is a crucial feature. The main issue with the previous state-of-the-art (SOTA) techniques is that they are not able to distinguish the visual appearance and margins of potholes close to the road areas. In order to achieve superior segmentation results and proper pothole margins, we have proposed an ERCU-Net model based on U-Net for pothole segmentation in this paper. The network consists of a novel residual convolutional block structure and employs a dropout layer in the residual convolutional block to minimize over-fitting concerns. A region-based IoU loss is proposed to reduce the regions of overlap between the anticipated segmentation and the actual segmentation. The performance of our proposed ERCU-Net model is compared with five SOTA CNNs in this paper. The result is also validated using Grad-CAM. The simulation results indicate that our proposed ERCU-Net model obtains the highest mean IoU 88.45% and the best overall pothole segmentation accuracy 98.55% in comparison to the existing state-of-the-art models.