Passive control may not provide enough damping for a stay cable since the control devices are often restricted to a low location level. In order to enhance control performance of conventional passive dampers, a new type of damper integrated with a rotary electromagnetic damper providing variable damping force and a flywheel serving as an inertial mass, called the rotary electromagnetic inertial mass damper (REIMD), is presented for suppressing the cable vibrations in this paper. The mechanical model of the REIMD is theoretically derived according to generation mechanisms of the damping force and the inertial force, and further validated by performance tests. General dynamic characteristics of an idealized taut cable with a REIMD installed close to the cable end are theoretically investigated, and parametric analysis are then conducted to investigate the effects of inertial mass and damping coefficient on vibration control performance. Finally, vibration control tests on a scaled cable model with a REIMD are performed to further verify mitigation performance through the first two modal additional damping ratios of the cable. Both the theoretical and experimental results show that control performance of the cable with the REIMD are much better than those of conventional passive viscous dampers, which mainly attributes to the increment of the damper displacement due to the inertial mass induced negative stiffness effects of the REIMD. Moreover, it is concluded that both inertial mass and damping coefficient of an optimum REIMD will decrease with the increase of the mode order of the cable, and oversize inertial mass may lead to negative effect on the control performance.