Scene depth information plays a fundamental role and can be beneficial to various computer vision or visual robotics applications. The scene color image acquired by consumer depth sensors usually has a high resolution, whilst its depth map counterpart often performs low resolution or man-made artifacts. Due to its strong similarity in terms of scene structures between RGB-D pairs, taking the color image as prior information, this paper proposes a Dual Branch Multi-scale Network (CA-DBMNet) based on the channel attention mechanism which can effectively guide the task of depth map super-resolution (SR). The network consists of two branches–color image feature extraction branch and depth map super-resolution branch. The first branch adopts the feature pyramid structure to extract the color image features, capturing image features and structures at different scales. The second branch is composed of three modules: 1) A dense residual feature fusion (DRFF) module to integrate the extracted features from two branches with dense connection and residual learning; 2) A channel multi-scale (CMS) module to exploit multi-scale features from depth feature maps; 3) A channel attention (CA) module to effectively enhance the channel proportion of high-frequency components in the depth feature maps. Extensive experiments demonstrate that CA-DBMNet can effectively reconstruct the high-resolution depth map with complete scene structures and sharp edges.