In the era of the Internet and the Internet of Things (IoT), more and more devices are using encryption technology to protect the privacy and reliability of communication data. Comparing the detection performance of malicious encrypted traffic before and after the improvement of the CNN model, the results showed that relying on the improved prevention method of CNN can effectively and quickly detect malicious traffic in encrypted traffic, improving the detection accuracy by about 0.0761, recall by about 0.0823, accuracy by about 0.0571, and has good robustness and generalization ability. The improved CNN model can more accurately locate malicious encrypted traffic, and has strong adaptability and recognition ability in the face of new security threats. The efficiency and accuracy of network security defense can be greatly improved, which can better ensure the security of important data, systems, and applications in the network.