Fall is one of the leading causes of injury and death for the elderly. Real-time fall detection is of great significance for the safety of the elderly. This paper proposes a coarse to fine fall detection algorithm based on Human characteristic matrix and Support Vector Machine (SVM). First, background subtraction and morphological processing are used to obtain more accurately human silhouette. Then, two human characteristic matrices are constructed based on Hu-moment invariant and the information of human body posture extracted from human silhouette and are used as features to train SVM classifier for fall detection. Experimental results indicate that the proposed algorithm can distinguish fall event from other movements such as squat, sitting down and back turning. Compared with other common methods, the proposed method can real-time and efficiently track the video with 18 frames per second.