The demand for high-speed low-power multi-modulus frequency divider is increasing in Phase-Locked Loop (PLL) design. In this paper, by combining the merits of traditional Johnson counter and Pulse-swallow frequency divider, we proposed a novel two-stage divider which can improve the operating frequency and decrease the power dissipation enormously. An adaptive component is built to set the divider in best power-saving mode. Based on the 40nm CMOS process, the frequency of this two-stage divider can reach 4GHz. The minimum power dissipation in divide-by-49 mode is 63µW@1GHz, or 156µW@4GHz. Compared with typical Johnson counter frequency divider, the frequency of the two-stage divider is improved about 1.6 times, while the power optimization ratio is 51.19%.