The effect of annealing temperature (TA) at 600–900 °C on structure and magnetic properties of the melt-spun Sm8.6Fe85.9Co4.3Zr1.2 alloys and its nitrides was investigated. Results show that the melt-spun powder is TbCu7-type SmFe9 single phase. X-ray diffractometer (XRD) Rietveld refinement shows that the lattice parameter a increases, c and c/a decrease with the increase in TA at 600–700 °C. This is due to the tendency of nitrogen entering into the 3f site (1/2, 0, 0) during the nitridation process, which is good for the magnetic properties. The phase transforms from TbCu7-type SmFe9 to Th2Zn17-type Sm2Fe17 at above 800 °C. Scanning electron microscopy (SEM) images show that the amount of surface defects of melt-spun powder which are harmful for magnetic properties decreases at first and then increases as TA increases. The coercivity (Hcj) and maximum magnetic energy product ((BH)max) of its nitrides increase first and then decrease as a function of TA. And the remanence (Br) always shows a decrease. The optimal magnetic properties of Hcj = 748 kA·m−1, Br = 0.92 T, (BH)max = 118 kJ·m−3 are obtained by annealing at 675 °C for 2 h and then nitriding at 460 °C for 10 h.