Scene classification is an important research content in computer vision, which is widely used in many fields, such as image and video retrieval, computer vision and robotics, monsters medical field and video surveillance. For the problem of video scene classification with complex background, how to ensure the accuracy of video feature extraction and classification is a challenge. In order to address this issue, taking the coal mine videos as example, this paper proposes a classification method of mine video scene based on improved Convolution Neural Network (CNNs). By increasing the depth of the original CNNs network, the algorithm improves the accuracy of feature extraction for complex background video. This whole network structure of this method consists of 10 layers of neurons. The first 7 layers are coiling, which are used to extract features, and the later 3 layers are all connected layers. The Softmax loss function in the full connection layer is used for classification. Experimental results show that the method proposed in this paper can effectively solve the problem of video scene classification with complex background.