Point matching is the basic input of many computer vision tasks. Eliminating outliers and mismatches to improve the accuracy of matching results is the ultimate goal of the point matching algorithm. In this paper, we propose a smooth point matching algorithm to find the robust point correspondences between image pairs. First, scale invariant feature transform (SIFT) feature points extracted from image pairs are selected as sample point sets, and we find the initial feature correspondence between the two point sets. Second, by combining the kernel function and the regularization term, we construct an effective form of energy function, and the optimal solution is found by the EM algorithm, so that the optimal mapping function is obtained. The final results of the experiment show that our proposed algorithm achieves better performance than the other algorithms in terms of precision and recall.