The network security protection of smart substation is the key to ensure the normal operation of substation. Firstly, the communication network architecture of smart substation and IEC61850 protocol stack are analyzed. The message authentication code function (Hash-MAC) designed by the Hash algorithm is used to encrypt the IEC61850 message safely, and the specific structure and implementation of the message security encryption in smart substation is given. Then, the dynamic key is introduced in the Hash-MAC calculation to reduce the risk of key cracking, and the difference sequence variance is introduced to detect abnormal packets of IEC61850 to improve the detection accuracy. Finally, the protective effect of the proposed method is verified by manufacturing attack experiments. The results show that the detection accuracy of the proposed algorithm is the highest for different types of messages, and the missed detection rate and average delay are smaller than other algorithms.