This paper presents a driver assistance system to monitor the driver driving behavior by applying the so-called “First-Person Vision” (FPV) technology. It consists of two modules: the scene classification and the driver viewing angle estimation. First, we use “bag of words” image classification approach based on FAST and BRIEF feature descriptor in the dataset. Second, we establish the “vocabulary dictionary” to encode an input image as a feature vector. Third, we apply SVM classifier to detect whether the driver's view is inside or outside scene of a vehicle. Finally, we estimate the driver viewing angle estimation based on FPV and the windshield-mounted camera. In the experiments, we illustrate the effectiveness of our system.