Sensitivity and figure of merit (FoM) are the most significant performance metrics for surface plasmon resonance (SPR)-based sensors. While designing an SPR-based sensor, one of the major concerns relies in maximizing the sensitivity and accuracy of the sensor. In most of the SPR sensor designs, not only there is observed a tradeoff between sensitivity and accuracy, but also these performance metrics are greatly influenced by different sensor parameters, such as choice and layer thicknesses of plasmonic metals, adhesive layers, add layers, field enhancement layers, and biorecognition element (BRE) layers along with type of light coupling prisms/glasses. In this article, rigorous analysis is carried out by considering most of the materials that are frequently used till date to achieve the optimized SPR sensor structure with high sensitivity and FoM. We have applied the inverse design approach utilizing the particle swarm optimization (PSO) method along with transfer matrix method (TMM) for this purpose. In this process, wide variety of frequently used sensor constituent layers are taken into account. The results indicate that a significantly high sensitivity (i.e., 630.54°/RIU) as well as FoM (i.e., 2277 RIU−1) can be achieved with the proposed method. The obtained results are closely compared with the sensor designs reported in the literature, and the comparison reveals a significant enhancement in both sensitivity and FoM. The findings of this study confirm the ample advantage and potential of utilizing this optimization approach for designing high-performance SPR sensors.