Stereo-radargrammetric shift estimation is an important part of interferometric synthetic aperture radar (InSAR) data processing. However, the presence of residual topographical phase poses a challenge to achieving accurate coherent shift estimation in future high-resolution InSAR measurement tasks. In this work, we present a two-stage multi-baseline InSAR stereo-radargrammetric shift joint estimation approach. Our proposed method reduces the influence of the residual topographical phase, even in cases where no prior information is available or with low resolution prior digital elevation models (DEMs). In addition, a topography model based on Brownian motion is used to analyze the effect of the residual topographical phase on the accuracy of the shift estimation.