Public transport vehicle scheduling system plays a very important role in relieving urban traffic pressure, and it's now still a difficult problem. In order to solve the problem effectively, we propose a bus scheduling method based on video processing algorithm in this paper. Firstly, we decode the H264 bare stream which receive from the bus in real-time, and according to the prior knowledge, a certain number of training samples with different density levels were collected, and then extract the improved gray-level co-occurrence matrix and color entropy features of training samples to train the classifier model. Finally, crowd density estimation result can get by processing the real time vehicle video using the trained model. Combined the result of density estimation with vehicle status information, we can realize the vehicle scheduling effectively.