In Vietnam, traffic is always a complex and challenging problem due to a mixture of different types of vehicle as well as the large number of vehicles on road. To improve the traffic management, it is critical to develop a real time traffic flow estimation system which can detect, classify and count vehicles, detect traffic violation at any given time. In this study, a multi-vehicle detection and tracking approach was proposed to achieve these requirements. The proposed model involved two major steps: detection and multiple-object tracking. In the first step, the vehicles were detected and classified into classes (motorbike, car, truck, bus and rudimentary vehicles) using Faster R-CNN model. Next, the movement of the detected objects was tracked with CSRT tracker. Then, all vehicle data is sent to an analyzer to estimate the traffic flow by counting and classifying vehicles by their driving direction. Results showed that the model can robustly work in realtime with an accuracy $ \gt 86$%.