Energy management strategies (EMSs) are crucial to the fuel economy of hybrid electric vehicles (HEVs). However, due to the lack of efficient solving approaches, most of existing EMSs mainly focus on the optimal torque split between the internal combustion engine (ICE) and the electric motor but neglect improper ICE on/off switches, and thus usually suffer degraded fuel economy and even unacceptable drivability in practice. To tackle this issue, this paper presents a novel EMS that uses an efficient actor-critic (AC) method to regulate ICE switches with limited computation resources. While common AC methods use complex neural networks (NNs) with arbitrary initialization, the proposed AC uses piecewise cubic polynomials whose parameters are initialized based on optimized solutions of dynamic programming (DP). By this means, the AC can quickly converge with high computation efficiency. The testing results from processor-in-the-loop (PIL) simulations showcase that, compared with a rule-based EMS with tabular value functions, the proposed EMS can greatly improve the equivalent fuel economy by eliminating improper ICE switches after only several iterations of adaptive learning and dramatically save the onboard memory space owing to the concise AC structure.