Since the safe and reliable behavior of a Railway Switch Machine (RSM) is definitely pivotal for rail transportation, scholars and engineers have studied a considerable number of algorithms for RSM fault diagnosis via many kinds of sensor information. However, scholars have yet to consider the impact of the sensors' states on RSM fault classification, which cannot be ignored in practical railway applications. To fill the gap, we propose an end-to-end deep learning architecture named Fault Diagnosis Considering Sensor Abnormality Network (FD-CSANet) for RSM behavior recognition under various sensor states. Our approach includes Sensor Information Aggregate Module (SIAM) and Fault Diagnosis Module (FDM). The SIAM utilizes the idea of the channel attention mechanism to filter and aggregate multiple sensor features, which weights normal sensor observation and weakens anomalous sensor information. The FDM adopts the DB EM as the backbone for the RSM fault diagnosis. Ultimately, ablation experiments and comparative studies demonstrate the superiority of our algorithm.