Track irregularity detection is crucial to assure passenger comfort and operation security of the train. In this paper, a novel track irregularity estimation model based on Temporal Convolution Network (TCN) and Long Short-Term Memory (LSTM) is proposed to monitor the track state promptly and accurately. Firstly, a denoising method combining Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) and grey theory is established to realize the noise reduction of the train attitude data with a poor signal-to-noise ratio (SNR). Then, the vehicle attitude characteristic information extracted from TCN is input into LSTM, and the complex model between train body lateral response and track irregularity is constructed. Finally, utilizing a dataset made up of real track irregularities and the vehicle-rail model, experimental validation is performed. The experiment results show that the track irregularity is detected effectively. Compared to the TCN, LSTM and CNN-LSTM, the detection error of TCN-LSTM is the lowest. The feasibility and effectiveness of the proposed method can be demonstrated.