The coexistence of the charge-density wave (CDW) and superconducting phases and their tunability under external pressure remains one of the key points in understanding the electronic structure of $A$V$_3$Sb$_5$ ($A$ = K, Rb, Cs) kagome metals. Here, we employ synchrotron-based infrared spectroscopy assisted by density-functional calculations to study the pressure evolution of the electronic structure at room temperature up to 17 GPa experimentally. The optical spectrum of CsV$_3$Sb$_5$ is characterized by the presence of localized carriers seen as a broad peak at finite frequencies in addition to the conventional metallic Drude response. The pressure dependence of this low-energy peak reflects the re-entrant behavior of superconductivity and may be interpreted in terms of electron-phonon coupling, varying with the growth and shrinkage of the Fermi surface. Moreover, drastic modifications in the low-energy interband absorptions are observed upon the suppression of CDW. These changes are related to the upward shift of the Sb2 $p_x+p_y$ band that eliminates part of the Fermi surface around the $M$-point, whereas band saddle points do not move significantly. These observations shed new light on the mixed electronic and lattice origin of the CDW in CsV$_3$Sb$_5$.