This paper deduces the small signal stability of AWS based wave power generation system. Because of the reciprocating movement of the rotor in the linear permanent magnet generator used in AWS, the small signal stability analysis is carried out when the rotor speed is greater than zero, less than zero or equal to zero. The results show that the system can keep small signal stability under these three conditions, and the characteristic roots are not much different. But some of these characteristic roots are close to the virtual axis. For this reason, set the distance between the real parts of the characteristic roots and the virtual axis as the objective function and the particle swarm optimization algorithm is used to optimize the controller parameters of the AWS based wave power generation system. The optimization results and the time domain simulation results show that the parameter optimization improves the system’s small signal stability.