Compared with single image super-resolution (SISR), reference-based image super-resolution (RefSR) utilizes additional references (Ref) to recover more realistic texture details, achieving better reconstruction performance. Most recent works focus on transferring relevant texture features from Ref to low-resolution (LR) images. However, those works ignore the high-frequency information existing in the LR space, leading to performance degradation when irrelevant Ref images are given. To address this issue, we propose a residual channel attention connection network for reference-based image super-resolution (RCACSR), which fuses valuable high-frequency information in LR space with high-resolution (HR) texture details of Ref. Specifically, the proposed residual channel attention connection network (RCACN) can extract more complex features from the LR space. Moreover, an enhanced texture transformer is presented, which can search and transfer texture features more accurately from Ref. Extensive experiments have demonstrated that the proposed RCACSR is superior to the state-of-the-art approaches in the aspects of both quantitative and qualitative measurements.