Synthetic Aperture Radar (SAR) images often suffer from inherent speckle noise that can significantly degrades their quality, and makes it difficult to identify important targets or extract useful information from them. To solve this problem, a novel SAR image despeckling model called HM-SIDLR is proposed, which can suppress noise and preserve image edges from low-rank residues simultaneously. Specifically, our method considers that there is some structural information discarded as noise in residues. Therefore, we construct a hierarchical model to extract more edge details which can compensate for the over-smoothing problem caused by removing most of the speckle noise in the low-rank part. So as to learn the information from the edge subspace, a tunable prior knowledge matrix is designed, allowing for differentiation between clean and noisy pixels. Experimental results with the Virtual SAR dataset indicate that HM-SIDLR can achieve comparable despeckling performance to the state-of-the-art (SOTA) results.