In order to solve problems of hand-eye calibration in robot vision systems, traditional methods are easy to be affected by noise in the calibration process, which leads to large errors. A new hand-eye calibration method based on hunter-prey optimization is proposed. The first step was to create the closed-loop mathematical model of robot hand-eye calibration, and error functions of the rotation part and the translation part of the hand-eye transformation matrix were respectively established by dual quaternion theory, and the constraint conditions were optimized by the outer point penalty function method. Then, the rotation optimization problem and the translation optimization problem were solved by the hunter-prey optimization algorithm. The problem of local optimal in the process of solving is avoided to the maximum extent. Finally, through numerical simulation and robot calibration experiments, the proposed method is compared with two traditional methods (classical two-step method Tsai and traditional dual quaternion method DQ). The results show that it has better solving accuracy and stability.