Information diffusion prediction aims to predict the tendency of information spreading in the network. Previous methods focus on extracting chronological features from diffusion paths and leverage relations in social graph as side information to facilitate diffusion prediction. However, abundant high-order social relations in information diffusion have not been sufficiently utilized, such as co-repose and co-following which can further mine potential user common preferences. In this paper, we construct a heterogeneous diffusion network (HDN) from the social graph and information cascades to model the high-order social relations in information diffusion. Then, we design a novel model named Multi-Channel Recurrent Graph Convolutional Network (MC-RGCN), which can extract high-order social relation semantics from the channels of HDN to promote prediction performance. In each channel, we depict a specific social relations from the views of global topology, pairwise strength, and local structure. Finally, we conduct extensive experiments on three real-world datasets, and the results show that our proposed method outperforms the state-of-the-art models on diffusion prediction.