Flyback converter has been widely used in small-to-medium power applications because of simple topology and few components. However, as the frequency of the switching mode power supply(SMPS) tends to be higher and the size of SMPS tends to be smaller, the electromagnetic interference(EMI), which is caused by the switching of power semiconductor devices, becomes more serious. The common mode(CM) noise source and propagation paths of flyback converter are analyzed, then the CM noise model of converter is established. A quantitative analysis method of incomplete shielding layer for CM noise suppression in flyback converter is proposed. The incomplete shielding layer is further divided into incomplete-width and incomplete-length shielding layer for different structural parameters. The x-y axis coordinate system of the transformer is established, then the expressions of electric potential distribution along windings are obtained. Based on the cancellation mechanism of CM displacement current, the calculation method of width or length of the incomplete shielding layer is given. Experimental results verify the effectiveness of proposed method and the accuracy of the calculation method.