The electrocardiogram (ECG) signal is vulnerable to being interfered with some unknown noises in the acquisition process due to their low frequency and amplitude, which leads to the loss of significant information in the signals. Recent deep learning models have achieved encouraging results in denoising, however, the generalization ability of the model is not robust to various noises, and the gradient difference between the denoised signal and the original signal is always ignored. In this paper, we propose a deep learning denoising method based on half instance normalization (HIN) block and gradient difference max (GDM) loss function, which includes two stages. In the first stage, we input the noisy ECG signal to obtain the denoised signal. In the second stage, we reconstruct the denoised signal by fusing the preliminary results of the first stage and correct the waveform distortion caused by the first stage denoising, to reduce the loss of information. A novel loss function is also proposed, which can consider the difference between the slope of the denoised ECG signal and the clean ECG signal. The experimental results on MIT-BIH databases show that our method reaches optimal performance both in signal-to-noise ratio (SNR) and root-mean-square error (RMSE).