The purpose of this paper is to study the nonlinear dynamics of a stay cable in cable-stayed bridges caused by vortex-induced vibration (VIV). The equation of the planar transverse motion of the stay cable under uniform wind is developed by using the van der Pol wake oscillator to simulate the wind load. The Galerkin method is applied to discretize partial differential equations into ordinary differential equations. Then, the method of multi-scale (MMS) is used to derive the modulation equations of the nonlinear coupling system with the primary resonance response, and the results are verified by the Runge–Kutta method. Parameters such as the damping ratio and the sag-to-span ratio of the stay cable, the aerodynamic parameters, and the coupling parameters on the frequency response are investigated. The results show that large-amplitude vibration occurs due to the high energy transfer between cable and wake in the lock-in region though the coupling effect of the system is very weak for low-order VIV. The structure parameters of the cable and the van der Pol parameters can have a great influence on the nonlinear dynamic response.