We present the UV-to-NIR size evolution of a sample of 161 quiescent galaxies (QGs) with $M_*>10^{10}M_\odot$ over $0.510^{10.6}M_\odot$. To constrain the physical mechanisms driving the apparent size evolution, we study the relationship between $R_e$ and the formation redshift ($z_{form}$) of QGs. For lower-mass QGs, this relationship is broadly consistent with $R_e\sim(1+z_{form})^{-1}$, in line with the expectation of the progenitor effect. For higher-mass QGs, the relationship between $R_e$ and $z_{form}$ depends on stellar age. Older QGs have a steeper relationship between $R_e$ and $z_{form}$ than that expected from the progenitor effect alone, suggesting that mergers and/or post-quenching continuous gas accretion drive additional size growth in very massive systems. We find that the $z>3$ QGs in our sample are very compact, with mass surface densities $\Sigma_e\gtrsim10^{10} M_\odot/\rm{kpc}^2$, and their $R_e$ are possibly even smaller than anticipated from the size evolution measured for lower-redshift QGs. Finally, we take a close look at the structure of GS-9209, one of the earliest confirmed massive QGs at $z_{spec}\sim4.7$. From UV to NIR, GS-9209 becomes increasingly compact, and its light profile becomes more spheroidal, showing that the color gradient is already present in this earliest massive QG.
Comment: 28 pages, 19 figures, submitted to ApJ