In this paper, we investigate the rolling interface instability characteristics caused by the coupled motion of strip and the roll system. Based on the experimental test, the dynamical behaviors of the rolling mill under the motion of roll system and the strip show multi-modal vibration phenomenon, which are consistent with the variation of rolling interface force and energy parameters. Then, the dynamic geometric parameters of rolling interface and the dynamic stress models are established. Through the simulation analysis, it can be concluded the multi-modal stress state of the rolling interface is caused by the coupling motion of the strip and the roll system. The results show that the multi-modal stress of the rolling interface cause the rolling process instability, leading the "pulling " on the strip and "beating" on the work roll, which change the instability state of the rolling process. This study provides a theoretical basis for the study of multi-modal vibration characteristics of rolling mill.