In the field of radio frequency (RF) device design, an optimization method of RF filter based on neural network is proposed in this paper, which is suitable for the rapid customized design of filter under different indexes. Compared with the traditional method, this optimization method avoids the complicated and tedious traditional process of using circuit theory to construct the model topology, and trial-and-error full-wave electromagnetic simulation, which significantly simplifies the optimal design of the filter. The optimization method of the filter response is performed by use of a fourth-order split resonant rings (SRRs) filter topology model operating at Ku-band with a center frequency at 12.75 GHz, a bandwidth of 0.7 GHz, and two transmission zeros at 11.2 GHz and 14.5 GHz. Finally, the optimization results are consistent with the expected index requirements.