2D to 3D pose lifting is a promising method in 3D human pose estimation, and exciting improvement has been achieved after Graph Convolutional Networks (GCNs) is introduced into this task. In this paper, we propose a training method that achieved symmetry constraints of the skeleton, and it works well combined with different lifting methods. To make full use of the connection relationship of the joints, we propose a novel GL-Net, which views the human skeleton as a graph, for lifting 2D pose to 3D. Then, we add a Body factor Net to extract features from the 2D human pose estimation networks for correcting the scale of the 3D skeleton. We validate our method on public datasets. Experiments show that our model makes great progress. The proposed lifting method should be a promising tool for accurate 3D human pose estimation.