Recently rumors have been rapidly propagated while the Internet has been extensively developed. Research shows that highly credible comments with a distinct stance have worthy information. In this paper, we attempt to combine user credibility and user stance to capture worthy comments during the information-dissemination process to detect rumors. We propose a User Stance Bi-Directional Graph Attention Networks (USBGAT) model to extract accurate information for rumor detection based on high credibility users with strong stance, and diminish ineffectively neutral comments. Specifically, we take user features and user stance as a component of the node features, with multiviews features of tweets content engaged. Then, we use bidirectional graph attention networks (GAT) to capture the high-level representation of the rumor. Furthermore, we reweight the node features according to users' stances. Extensive experiments on two datasets: Pheme and Weibo show that our model is superior to the state-of-the-art models, especially in the early rumor detection. Our code and data are available at https://github.com/TAN-OpenLab/USB-GAT