Numerical simulations on the aerodynamics of helicopter ducted tail rotor in hover are conducted, and a surrogate-based optimization framework is employed for the duct shape design in this work, by considering interactions between rotor and duct. The CFD analyses of ducted tail rotor are performed using the helicopter code CLORNS, including the momentum source method and high-fidelity CFD methods. The SA365N1 Dauphin fenestron in a test case is simulated by CFD methods and the duct shape is selected as baseline design for optimization. Parametric researches are conducted to investigate the relationship between the aerodynamic loads of ducted tail rotor and the variations of duct shape. Besides, the influence of duct shape on the flow details of rotor blade is obtained by resolving the flowfield based on high-fidelity CFD, and detailed analyses of the flow separation phenomenon downstream the rotor with large collective pitches are presented. Then, the multi-optimization of duct shape with different collective pitches is conducted by employing a surrogate-based optimization framework. Compared with the baseline design, the hovering efficiency of optimized design is improved because of a better negative pressure distribution on the duct wall.