Heterodyne efficiency is an indicator to evaluate the performance of space coherent laser communication systems. It is affected by signal light and local oscillator (LO) light amplitude, phase and polarization state. In this paper, based on the common heterodyne efficiency, a heterodyne efficiency model that can reflect polarization aberration of optical system is proposed. The heterodyne efficiency is analyzed when the signal light and the LO light are linearly polarized or circularly polarized. For a coherent communication optical system, when the incident signal light is right-circularly polarized light and the incident LO light is 45° linear polarized light. Based on the three-dimensional ray tracing theory and the heterodyne efficiency proposed in this paper, the change of polarization states and the distribution of heterodyne efficiency of the signal light and LO light influenced by the optical system’s polarization aberration are analyzed. Analysis shows that the heterodyne efficiency model proposed in this paper can be used to evaluate coherent communication systems and reflect the influence of optical system polarization aberration.