In a free-space coherent optical communication system, wavefront distortion is frequently beyond the correction range of the adaptive-optics system after the laser has propagated through the atmospheric turbulence. A method of residual wavefront correction is proposed, to improve the quality of coherent optical communication in free space. The relationship between the wavefront phase expanded by Zernike polynomials and the mixing efficiency is derived analytically. The influence of Zernike-polynomial distortion on the bit-error rate (BER) of a phase-modulation system is analyzed. From the theoretical analysis, the BER of the system changes periodically, due to the periodic extension of wavefront distortion. Experimental results show that the BER after correction is reduced from 10−1 to 10−4; however, when the closed-loop control algorithm with residual correction is used, the experimental results show that the BER is reduced from 10−1 to 10−7.