With the rapid development of the Internet, network security and related technologies have become a topic of public concern. A new network architecture—SDN, began to attract widespread attention. The SDN network architecture has the characteristics of centralized control and programmability, which can obtain the network topology and traffic information of the entire network. At the same time, the SDN network can manage the entire network at the control layer and take countermeasures at the control layer to respond to network attacks quickly. However, due to the open nature of SDN, SDN network is also more vulnerable to cyberattacks.To solve the problem of SDN network security, this paper proposes a machine learning algorithm of LightGBM combined with the CNN algorithm to realize network abnormal traffic detection and achieve a good classification effect. Firstly, the LightGBM is used to identify abnormal network traffic, and the model can process large data volumes efficiently, which makes the proposed method applied to the security assurance of large-scale SDN networks effectively. At the same time, the CNN algorithm is used to classify the identified network abnormal traffic. Using a 10% subset of the KDD CUP 99 dataset for experimental verification, this combined machine learning algorithm can effectively identify 22 types of abnormal traffic, effectively improving the recognition accuracy.