This research proposed an algorithm conducting context awareness and tracking of a moving object. The research process is as follows: First, this study carried out preprocessing, such as binary coding, noise elimination and opening-closing arithmetic operations, after acquiring background subtraction image using BSM for moving object extraction. Second, this research eliminated the shadow of a moving object using three characteristics, namely chromatic image, brightness change and shadow direction in order to eliminate shadows, which a negative factor for image analysis. Third, this research calculated the skeleton line of a moving object by applying the thinning algorithm. Then, this research perceived the rapidly changing context of the moving object using the slope of the skeleton line. Lastly, the center point of the moving object was found by applying the pruning algorithm to the skeleton line, while it created moving direction and moving trajectory by comparing each frame central point of the moving object. As a result of the experiment, the rapid movement change of a moving object was extracted, and the moving direction was found through a central point comparison, while the method proposed by creating a moving trajectory was confirmed to be valid.