We study the valence crossover and strong electron-phonon coupling of EuPd$_2$Si$_2$ by polarization-resolved Raman spectroscopy. The fully-symmetric phonon mode shows strongly asymmetric lineshape at low temperature, indicating Fano-type interaction between this mode and a continuum of electron-hole excitations. Moreover, the frequency and linewidth of the phonon modes exhibit anomalies across the valence-crossover temperature, suggesting the coupling between valence fluctuations and lattice vibration. In particular, two phonon modes show significantly enhanced Gr\"uneisen parameter, suggesting proximity to a critical elasticity regime. The relative contribution of the structural change and valence change to the phonon anomalies is evaluated by density functional theory calculations.