This paper is using machine vision technology, combined with a coal gangue sorting algorithm and carried out in-depth research. In the aspect of coal gangue image segmentation and classification recognition, the fast segmentation algorithm based on video morphology and the classification and recognition algorithm based on support vector machine are respectively adopted. With the addition of feature extraction and a large number of experimental data, the sorting accuracy of the algorithm has been significantly improved.This article searches for datasets containing coal mine image data or coal mine classification labels in Kaggle, UCI machine learning libraries, etc.The experimental results show that the algorithm can process the multi-dimensional information and the corresponding color, shape and texture characteristics of coal gangue, optimize the separation effect, and effectively realize the fast and accurate separation of coal gangue. It is found that the coal gangue sorting algorithm based on machine vision has multi-field capability and can be widely used in coal mines and other fields, and has certain practical value and market prospect.