Ultrafast manipulation of vibrational coherence is an emergent route to control the structure of solids. However, this strategy can only induce long-range correlations and cannot modify atomic structure locally, which is required in many technologically-relevant phase transitions. Here, we demonstrate that ultrafast lasers can generate incoherent structural fluctuations which are more efficient for material control than coherent vibrations, extending optical control to a wider range of materials. We observe that local, non-equilibrium lattice distortions generated by a weak laser pulse reduce the energy barrier to switch between insulating and metallic states in vanadium dioxide by 6%. Seeding inhomogeneous structural-fluctuations presents an alternative, more energy efficient, route for controlling materials that may be applicable to all solids, including those used in data and energy storage devices.