In order to improve the resolution and the measurement accuracy of optical fiber sensor based on Fabry-Perot (FP) interference, the demodulation algorithm for the FP interference spectrum is analyzed in this paper. Signal demodulation algorithm directly affects the performance of the FP sensor, such as resolution and measurement accuracy. Combined with the basic structure of the sensor, a high-resolution vernier demodulation algorithm is studied based on fast Fourier transform (FFT) algorithm and minimum mean square error algorithm, which has a resolution of 0.084nm according to the experimental results. Then, the demodulation algorithm is applied to the FP interference optical fiber pressure sensor with stainless steel diaphragm. The test results show that the performance is good in the range of 0-1MPa with the resolution of 2.10×10 −4 MPa and correlation coefficient of 0.99989.