Fundus images are commonly used to document the presence and severity of various retinal degenerative diseases, where the fovea, optic disc (OD), and optic cup (OC) serve as important anatomical landmarks. Locating and segmenting these landmarks are crucial for clinical diagnosis and treatment. Many existing methods treat the recognition of the fovea, OD, and OC as separate tasks without incorporating any clinical prior knowledge related to various anatomical structures. In this paper, we propose a prior information guided coarse-to-fine dual-branch encoding network, which enables fovea localization and OD/OC segmentation. In coarse stage, we employ a dual-branch network consisting of convolutional neural network (CNN) and Transformer to encode local and global features, and then utilize multi-scale feature fusion techniques to merge the extracted semantic features, aiming to enhance the localization accuracy. In addition, we effectively use the distance information from each pixel to the landmark of interest, and output the results of distance map and heat map regression as prior information to further guide the network to learn the positional relationship between fovea and OD. In fine stage, we refine the region of interest (ROI) of the OD, balance the distribution of the OD and OC using polar coordinate transformation (PCT), extract critical boundary features using the boundary attention module (BAM), and improve the generalization performance of our method through model ensemble strategy. Extensive experimental results demonstrate that our proposed method outperforms existing state-of-the-art (SOTA) methods on the publicly available GAMMA and REFUGE datasets.