The visual system is the most important sensory organ of human beings. Most of the external information obtained by human beings comes from vision. With the continuous exploration of the natural world by human beings, people always hope to realize the function of human perception through some digital machine and automatically acquire the information of the outside world. The traditional CamShift algorithm needs to manually locate the target during tracking, and it can be used in complex backgrounds such as color interference and occlusion. In this paper, a comfortable and autonomous swimmer tracking algorithm based on Kalman filter with CamShift is proposed. First, we use the Canny edge detection and inter-frame difference method to segment the entire region of the moving swimmer and then use the extracted target region to initialize the CamShift algorithm. The modified search window supports the automatic tracking of the swimmer. When there is similar color interference in the background, or the swimmer is severely impeded, an improved algorithm combined the Kalman filter with the CamShift algorithm for tracking the target accurately. Experimental results confirmed that the enhanced algorithm significantly improves the outcomes and it can still follow the object effectively and steadily under severe occlusion, noisy environment and color interference.