For the safe and economic operation of new power system comprising large-scale PV power generation, a highly accurate short-term photovoltaic (PV) power forecast is of significance. Currently, traditional statistical indicators are utilized to evaluate the prediction results, and it lacks a more reasonable comprehensive evaluation system to assess the merits and demerits of the selected prediction model and system. Consequently, this paper firstly proposes a series of comprehensive evaluation index system that includes common error evaluation indexes, forecast assessment indexes, and volatility indexes, considering the forecast assessment indexes of prediction results and the impact of cloud motions on PV power in PV sites. Then, a thorough evaluation approach based on technology for order preference by similarity to ideal solution (TOPSIS) for PV power prediction (PVPP) errors is developed. Finally, based on actual data from a PV power plant, various prediction methods and evaluation indexes are assessed comprehensively, which confirm the feasibility of the proposed evaluation approach of PVPP errors.