This paper presents a robotic measurement system for the precise double-sided inline inspection of optical elements. The robotic system includes an electromagnetically actuated measurement platform (MP), capable of inducing a stiff link between the integrated high precision confocal chromatic sensor (CCS) and the sample by means of feedback control. In this way, disturbing relative motion between the CCS and the sample are compensated, establishing the desired lab-like conditions during the point-wise acquired 3D measurement. The high-precision positioning capability of the MP is used to precisely move the CCS’ measurement spot across the sample surface in a tailored scan pattern. The robotic 3D measurement system provides a lateral measurement area of $4\times 4\ \mathbf{mm}^{2}$ and achieves a lateral and axial resolution of $5\ \mu \mathrm{m}$ and 130 nm, respectively. In addition, high-precision 3D measurements of arbitrary regions of interest can be performed. Experimental results of a commercial lenslet array demonstrate, that the robotic sample-tracking system increases the double-sided surface and thickness measurement performance by at least one order of magnitude compared to the state-of-the-art approach, while enabling the detection of surface defects on the sub-micrometer scale.