This paper presents a moire fringe signal detection and processing system based on improved particle swarm optimization (PSO) algorithm, which can effectively improve the subdivision accuracy of photoelectric encoder. Firstly, based on the working principle of photoelectric encoder and the generation mode of Moire fringe signal, the Moire fringe signal is expressed by constructing a mathematical model. And the subdivision error of Moire fringe signal is analyzed, and the correction model of signal error is constructed. By analyzing and comparing different algorithms, the particle swarm optimization algorithm is selected to solve the parameters of the actual moire fringe signal, and the particle swarm optimization algorithm is optimized to reduce the number of iterations and speed up the operation speed. By drawing the normal distribution of the number of iterations before and after the improvement, the effect of the improvement of the algorithm can be observed. The designed system can compensate and correct the factors of subdivision error caused by quadrature deviation, DC level drift and sinusoidal error.