We include the treatment of quadrupolar fields beyond the Fr\"ohlich interaction in the first-principles electron-phonon vertex in semiconductors. Such quadrupolar fields induce long-range interactions that have to be taken into account for accurate physical results. We apply our formalism to Si (nonpolar), GaAs, and GaP (polar) and demonstrate that electron mobilities show large errors if dynamical quadrupoles are not properly treated.