In order to reduce the end effect of permanent magnet linear synchronous motor (PMLSM) on thrust ripple, a hybrid algorithm based on topology optimization is proposed to optimize the end teeth of PMLSM. Compared with the traditional geometry parameter optimization scheme, the hybrid optimization algorithm based on Tabu search and ON/OFF method is more stochastic and global. In optimization process, the detent force of the motor is analyzed by finite element method. Through post-processing of the optimized end teeth shape, a new feasible topology structure is obtained. The results show that the thrust ripple of motor is reduced by 13% after topology optimization, which proves the effectiveness of the method.