Color image is a two-dimensional projection of a three-dimensional scene, the image in the process of acquiring the depth information of the scene is lost, so image segmentation algorithms relying solely on color information are often difficult to achieve the ideal segmentation results. In this paper, we propose an image segmentation algorithm that incorporates depth information. On the basis of the segmentation results of the color map, evaluates the segmentation results of the color image by using the good edges of the depth image and the clustering of the pixels, identifies the regions with segmentation errors, and then conducts a secondary segmentation of the segmentation error regions by using the grayscale clustering and color-independence of the depth map in order to correct the segmentation results. Compared with the traditional color image segmentation, this algorithm can achieve better segmentation results.