Velocity dispersion ($\sigma$) is a key driver for galaxy structure and evolution. We here present a comprehensive semi-empirical approach to compute $\sigma$ via detailed Jeans modelling assuming both a constant and scale-dependent mass-to-light ratio $M^*/L$. We compare with a large sample of local galaxies from MaNGA and find that both models can reproduce the Faber-Jackson (FJ) relation and the weak dependence of $\sigma$ on bulge-to-total ratio $B/T$ (for $B/T\gtrsim 0.25$). The dynamical-to-stellar mass ratio within $R\lesssim R_e$ can be fully accounted for by a gradient in $M^*/L$. We then build velocity dispersion evolutionary tracks $\sigma_{ap}[M^*,z]$ (within an aperture) along the main progenitor dark matter haloes assigning stellar masses, effective radii and Sersic indices via a variety of abundance matching and empirically motivated relations. We find: 1) clear evidence for downsizing in $\sigma_{ap}[M^*,z]$ along the progenitor tracks; 2) at fixed stellar mass $\sigma\propto(1+z)^{0.2-0.3}$ depending on the presence or not of a gradient in $M^*/L$. We extract $\sigma_{ap}[M^*,z]$ from the TNG50 hydrodynamic simulation and find very similar results to our models with constant $M^*/L$. The increasing dark matter fraction within $R_e$ tends to flatten the $\sigma_{ap}[M^*,z]$ along the progenitors at $z \gtrsim 1$ in constant $M^*/L$ models, while $\sigma_{ap}[M^*,z]$ have a steeper evolution in the presence of a stellar gradient. We then show that a combination of mergers and gas accretion are likely responsible for the constant or increasing $\sigma_{ap}[M^*,z]$ with time. Finally, our $\sigma_{ap}[M^*,z]$ are consistent with a nearly constant and steep $M_{bh}-\sigma$ relation at $z\lesssim 2$, with black hole masses derived from the $L_X-M^*$ relation.
Comment: MNRAS accepted, 22 pages, 17 figures