Automated object tracking in video sequences has been applied in areas such as security and surveillance, traffic control, medical image processing and video communications. To maintain robust precision in matching our algorithm’s position estimates with ground truth, feature detection accuracy and confidence measures are needed by tracking algorithms. Experiments are conducted to quantify how feature descriptors used for tracking are degraded. A Kalman filter is then used to enhance accuracy. In our application, Kalman covariance parameters are continually tuned using a confidence level obtained based on object descriptor robustness. Because the Kalman algorithm converges quickly and does not require prior training, it is ideally suited for real-time object tracking.