Increasingly stringent national emission standards for construction machinery and rising fuel prices have promoted excavator manufacturers to improve energy efficiency. When the hydraulic excavator boom lowers, the gravitational potential energy is converted into heat, which results in poor energy efficiency. To solve this problem, a novel flywheel based energy recovery system (FESS) is proposed in this paper. A hydraulic pump/motor (PM) is employed as the energy conversion element and a flywheel is used as the energy storage element. To reduce the power loss when the flywheel is idling, a clutch is installed between the PM and the flywheel. The working principle of the system is analyzed in detail. A 4 t hydraulic excavator is taken as the research object, and the key parameters of the system are matched. The model of the FESS is developed in AMESim. The influence of the key parameters such as the PM displacement and the moment of inertia of the flywheel on the energy recovery and reutilization efficiency in the typical working cycle are simulated and analyzed, and the specific value of the energy recovery and reutilization is obtained, up to 65% for 4 t excavator and 60% for a 21 t excavator. The simulation results show that the FESS can significantly improve the energy efficiency of the hydraulic system.