Nonaxisymmetric endwall is an effective method to reduce secondary loss and improve aerodynamic performance. In this paper, a nonaxisymmetric endwall automated optimization process based on the nonuniform rational B-spline surface (NURBS) technique was proposed. This technique was applied for the aerodynamic optimization of the turbine stator shroud endwall to reduce total pressure loss and secondary kinetic energy. The flow fields of the datum endwall design (Datum) and optimization endwall design (Opt) were investigated and compared. Quantitative loss analysis was performed with a loss breakdown method. The entropy generation was classified as profile loss, secondary loss and trailing edge loss, all of which were reduced. The secondary loss was much smaller than the profile loss. In general, the blade row total entropy loss decreased by 11.7%. The results showed that the Opt design reduced total pressure loss and coefficient of secondary kinetic energy by 11.1% and 11.0%, respectively. The decrease in secondary kinetic energy could be attributed to the reduction in the horseshoe vortex and the reduced transverse pressure gradient. When the outlet Mach numbers and inlet incidence angles vary, the performance of the profiled endwall design was always better than the datum design. In the turbine stage simulation, the efficiency was increased by 0.28% with nonaxisymmetric endwall.