The problem of nonlinear combustion instability seriously affected the solid rocket motors and has been widely concerned by researchers. Experimental studies have shown that there is a clear multi-order modes coexistence phenomenon in nonlinear combustion unstable pressure oscillation, and there is energy transfer between modes. Therefore, in order to study the evolution of energy transfer between modes in nonlinear combustion instability, modes extraction of characteristic signal data is required. In this paper, based on VMD decomposition method, considering that artificially set parameters required by VMD decomposition have great influence on the results, a mode decomposition method SPSO-VMD is proposed for nonlinear unstable characteristic signals of SRM, and the typical nonlinear unstable signal is extracted and analyzed. The results show that the proposed method can accurately obtain the frequencies and amplitudes of various modes in nonlinear oscillating signals.