In this paper, we present physics-informed graph residual learning (PhiGRL) to model the scattering of 3D PEC targets by solving combined field integral equations (CFIEs). Emulating the computing process of the fixed-point iteration method, PhiGRL iteratively modifies the candidate solutions of CFIEs regarding the residuals of CFIEs until convergence. In each iteration, the matrix-vector multiplication of CFIE is incorporated to guide PhiGRL. The graph neural networks (GNNs) are applied to deal with the unstructured discretization and varying unknown numbers. With the data set generated by the method of moments (MoM), PhiGRL is first trained to model the scattering of basic 3D PEC targets, including spheroids, conical frustums, and hexahedrons. Furthermore, the transfer learning strategy is adopted to migrate PhiGRL to simulate airplane-shaped targets. Numerical results validate that PhiGRL can provide real-time and accurate simulations of 3D PEC targets. This study explores the feasibility of combining deep learning and physics to accelerate the 3D EM modeling.