Micro-displacement measurement is a crucial task that should be meticulous from construction examination to human vital sign detection. Among quantities of approaches, frequency-modulated continuous-wave (FMCW) radar is characterized by its portability and inexpensiveness. However, for accurate measurement, one must face clutter that would be troublesome for processing. In this paper, we reform each chirp signal into sub-groups, exploiting their geometrical properties in the IQ domain, and propose the Chirp Grouping and Multi-Circle Fitting (CG-MCF) method which further improves the precision of measurements. Simulation and experiments are given, proving its validity and stability at the displacement amplitude of as even low as 10 to 20.