In this paper, AlN films grown by magnetron sputtering method have been proposed as the gate insulator layer of AlGaN/GaN high electron mobility transistors (HEMTs) to decrease gate leakage current and suppress the interface trap. The effect of the temperature of substrate on the quality of AlN films have been investigated. By inserting the thin AlN film (35 nm) as a gate insulator layer, the on-state resistance of AlGaN/GaN HEMTs decrease from 11.1 Ω mm to 10.3 Ω mm @ V g = 0 V, the current collapse decreases from 16.6% to 3.2%, the gate leakage can be reduced from 1.2 Ă— 10â'1A mmâ'1 to 4.4 Ă— 10â'6A mmâ'1 @ V g = 2 V by five orders of magnitude, and the fast interface states disappear and the normal trap density decreases from 0.96â€"1.3 Ă— 1013 cmâ'2eVâ'1 to 1.3â€"3.4 Ă— 1012 cmâ'2 eVâ'1, proving that magnetron-sputtered AlN is an effective way to improve the performance of GaN HEMTs. [ABSTRACT FROM AUTHOR]