A Railway Point Machine (RPM) used to switch alternative routes for trains, is critical equipment in the railway industry since the states of RPMs directly affect the efficiency of rail transportation and even rail safety. Therefore, diagnostics of RPMs are indispensable. A proposed method for robust diagnostics of RPMs via sound signals, considering both robustness and non-contact aspects. This is the first time that we have overcome the limitations of ideal environment by adding noise to get closer to the real scene. Firstly, we collect sound signals from the test rig, and then we add multiple noises randomly and dynamically for each piece of data. Finally, Convolutional Neural Network (CNN) is used for training. The experimental results show that compared with Support Vector Machines (SVMs) and K-Nearest Neighbors (KNNs), our method has superior robustness with guaranteed accuracy (at 95.86%).