In this work, the HZO-based ferroelectric (FE) polarization switching speed is experimentally and systematically investigated considering device structure and fabrication process. The mechanisms of FE switching speed dependence are further analyzed, and the effect on nucleation process during switching is found to be the dominant factor. Furthermore, design guidance for high-speed low-power FE devices on device structure and annealing condition is proposed based on the discussion, paving the way for the design of high-performance FE devices and circuits.