To detect and analyze malicious traffic accurately, a malicious traffic detection system is implemented by using machine learning algorithms and visualization methods. The system is divided into three modules: the traffic detection module builds the traffic detection model through the random forest algorithm and uses the CIC-IDS-2017 dataset to train and optimize the model. The Traffic Capture and Feature Extraction module uses sniffer technology to capture the traffic packets flowing through the network card and extract the characteristics of the traffic data. The visualization module uses B/S architecture to provide data visualization support after traffic detection. To illustrate the system's specific design, the basic process of the intrusion detection system will be used. An example analysis shows that in the actual network environment, the model can detect malicious traffic types more accurately after training the dataset and re-optimizing the real traffic. At the same time, the system can visualize the display and e-mail warning based on the detection results, improve user-friendliness, and provide support for in-depth analysis of malicious traffic.