Roads in India do not always maintain a smooth surface due to the presence of potholes, cracks, and eroded surfaces which may pose a serious threat to the safe and comfortable travel of road commuters and the operating condition of commuting vehicles. Among the ones listed, the potholes are riskier as they may cause steering misalignment, wheel damage, and engine damage in commuting vehicles and lead to accidents. An accurate fast detection of potholes is essential and pre-emptive to achieve comfortable travel for the passengers, especially in advanced driver assistance systems (ADAS)-enabled vehicles. In this work we have used two object detection algorithms (ODAs), you only look once (YOLO)v5s and Faster regions with convolutional neural network (Faster RCNN) to detect the presence of potholes on road surfaces from the images of road surfaces. The two ODAs are trained, validated, and tested using a curated dataset of 4097 images which are obtained from open source and annotated for the class ‘pothole’. The performance of the two models is evaluated on the basis of mean average precision (mAP) and the time taken for processing and detection of potholes. The two models are trained with different epochs values as 50, 100, and 150 and the test results obtained by testing 383 images show that for 150 epochs, Faster RCNN is better in terms of mAP value than YOLOv5s. On the other hand, while running on Google Colab Pro, YOLOv5s has inferred test images in 1.7 minutes compared to Faster RCNN, which takes a detection time of 16 minutes