In order to improve the association matching of multiple pedestrians in occlusion scene, an improved algorithm is proposed in this paper that combines the similarity matching of pedestrian object’s major color spectrum histogram combination with joint probabilistic data association. The similarity to major color spectrum histogram of pedestrian’s object is calculated between the previous and the frame that current frames in this algorithm. And take the centroid location of pedestrian’s target which is with the most largest similarity to correct the state vector of the JPDA algorithm in the current frame. The experimental results show that the proposed algorithm could make the number of the targets’ label adhesion reduced to 40%, the number of the targets’ label confusion decreased about 33%, the recall rate of targets is increased by 6.8%, and the tracking accuracy is improved by 9.2% compared with the classical JPDA algorithm when the occlusion rate between pedestrians and intensity are not very high.