In many methods, before inputting Electroencephalography (EEG) signals into the Convolutional Neural Network (CNN), the signals are usually preprocessed by low-pass filter, wavelet threshold denoising and so on. In this paper, it is found that these operations can be completely replaced by CNN, which automatically ignores noise interference in the recognition training process. A light and effective network named Shallow Residual CNN (SRCNN) is proposed to recognize the EEG hand-motion signals whose dataset is from Kaggle competition. The experiment results indicated that our proposed method can directly manage the raw data and have a better performance under ROC curve area evaluation when comparing with the classic deep learning network ResNet 34. This paper can be a good example to extend the CNN-based scheme to more types of EEG signal.