Based on manifold learning (ML), a novel driver behavior recognition (DBR) method is proposed in this paper to recognize the lane-changing behaviors of surrounding vehicles based on the camera-only information. In our study, one of the most widely-used ML methods, isometric mapping (Isomap), is adopted to find the latent manifold structure of the driving data extracted from real-world video frames. Based on the manifold found, the support vector machine (SVM) is applied as a classifier to classify the lane-changing process into three different phases, namely ‘before lane change’ (BLC), ‘lane change’ (LC) and ‘after lane change’ (ALC). After training, different phases can be recognized by SVM. To test the performance of the proposed method, experiments using real-word data are designed and carried out. A linear dimensionality-reduction method, principal component analysis (PCA), is used for comparison. The experimental results verify the ability of ML for finding the low-dimensional structure of data, and compared with PCA, SVM with Isomap shows better performance on the prediction accuracy.