With the development of multimedia and communication technology, the amount of multimedia data, for example, video data has been rapidly emerging. How to process these data accurately and efficiently has attracted more and more researchers' attention. Video classification is an important part of video data processing. After deep learning method successfully applied in image and audio fields, the study of video classification has gradually shifted to deep learning direction. This paper summarized the video classification methods based on deep learning and analyzed the differences in performance of typical algorithms, meanwhile, summarized the commonly used video classification data sets.