In this paper, we focus on blind super-resolution, which is a challenging problem due to the unknown degradation process. Previous methods assume a simplified degradation model and fail to generalize to multiple real-world scenarios. We reformulate the image degradation with a more general assumption and embed it as a degradation latent vector. To predict the latent, we first construct a correlation graph between input LR images and pre-sampled LR-HR image pairs. And then through a graph propagation, the degradation latent is obtained, aggregating valuable information from the graph. We name the graph module as GALE (Graph-bAsed Latent Estimation), which is referred as a connection to the past. Furthermore, we design a novel blind SR framework GALE-SR. Extensive experiments on synthetic and real-world images show that the proposed GALE-SR can provide visually favorable and state-of-the-art performance in blind SR problem.