The central quantity in the celebrated quantum Jarzynski equality is $e^{-\beta W}$, where $W$ is work and $\beta$ is the inverse temperature. The impact of quantum randomness on the fluctuations of $e^{-\beta W}$ and hence on the predictive power of the Jarzynski estimator is an important problem. Working on a single nitrogen-vacancy center in diamond and riding on an implementation of two-point measurement of non-equilibrium work with single-shot readout, we have conducted a direct experimental investigation of the relationship between the fluctuations of $e^{-\beta W}$ and adiabaticity of non-equilibrium work protocols. It is observed that adiabatic processes minimize the variance of $e^{-\beta W}$, thus verifying an early theoretical concept, the so-called principle of minimal work fluctuations. Furthermore, it is experimentally demonstrated that shortcuts-to-adiabaticity control can be exploited to minimize the variance of $e^{-\beta W}$ in fast work protocols. Our work should stimulate further experimental studies of quantum effects on the bias and error in the estimates of free energy differences based on the Jarzynski equality.