When the behavior recognition model is used to recognize the violation behavior of underground operation, the surface characteristics of the skeleton sequence are mainly obtained by the simple feed-forward neural network, which leads to the low mAP value of the recognition result. In order to solve this problem, deep learning technology is integrated into the process of behavior recognition, and a new method of underground operation violation recognition is designed. Firstly, the image data of underground operation behavior is collected, and the data is enhanced by the CutMix algorithm. And then a three-dimensional skeleton sequence of that human body is established. The running data of the skeleton distance and the skeleton angle are calculated, and the visual angle in-variance characteristic of the human body skeleton is obtained. Finally, the convolution neural network is used to build a behavior recognition model based on deep learning, extract the surface and deep Spatio-temporal fusion features of the skeleton sequence, and output the violation identification label. The experimental results show that the mAP value of the proposed method is 0.93, which effectively improves the identification accuracy and can better guide the safety management of underground operations.