Zone-center optical phonon linewidth is a key parameter for infrared and Raman spectra as well as the Lorenz oscillator model. While three-phonon scattering was often assumed to be the leading contribution, in this work we find, surprisingly, that higher-order phonon scattering universally plays a significant or even dominant role over three-phonon scattering at room temperature, and more so at elevated temperatures, for a wide range of materials including diamond, Si, Ge, boron arsenide (BAs), cubic silicon carbide (3C-SiC), and $\alpha$-quartz. This is enabled by the large fourphonon scattering phase space of zone-center optical phonons, and distinct from heat conduction where at room temperature four-phonon scattering is still secondary to three-phonon scattering. Moreover, our results imply that five-phonon and even higher-order scattering may be significant for some large band-gap materials, e.g., BAs. Our predicted infrared optical properties through the Lorenz oscillator model, after including four-phonon scattering, show much better agreement with experimental measurements than those three-phonon based predictions. This work will raise broad interest of studying high-order scattering in various areas beyond heat conduction.