Thermal switching materials, whose thermal conductivity can be controlled externally, show great potential in contemporary thermal management. Manipulating thermal transport properties through magnetic fields has been accomplished in materials that exhibit a high magnetoresistance. However, it is generally understood that the lattice thermal conductivity attributed to phonons is not significantly impacted by the magnetic fields. In this study, we experimentally demonstrate the significant impact of phonon-magnon scattering on the thermal conductivity of the rare-earth metal gadolinium near room temperature, which can be controlled by a magnetic field to realize thermal switching. Using first-principles lattice dynamics and spin-lattice dynamics simulations, we attribute the observed change in phononic thermal conductivity to field-suppressed phonon-magnon scattering. This research suggests that phonon-magnon scattering in ferromagnetic materials is crucial for determining their thermal conductivity, opening the door to innovative magnetic-field-controlled thermal switching materials.