In recent multi-speaker speech separation researches, the overall deep-learning-based architecture consists of three parts: encoder, separator, and decoder. But improvement strategies generally only focus on the separator in the middle, regardless of its input. The most common encoder structure at present is a single 1D convolution layer followed by a nonlinear activation function, ReLU. In this paper, we firstly propose a new encoder named Attention DE, trying to improve the input effectiveness of the separator. The new encoder adds extra 1D convolutional layers and the multi-head attention mechanism to enhance the feature aggregation ability of input speech. Secondly, instead of RNNs, our separator uses SepFormer Blocks to improve the training efficiency and learn the speech sequence patterns better. Experiments show that the Attention DE is generally applicable to improve the performance of the single-channel speech separation model based on the time domain. The method of Attention DE fusion SepFormer blocks achieves an advanced SI-SNRi of 20.3dB on WSJ0-2MIX. Code is publicly available at https://github.com/TAN-OpenLab/AttentionDE.