We study nonlinear resonance of coupled modes in nano-mechanical systems. To reveal the qualitative features of the dynamics, we consider the limiting cases, where the results can be obtained analytically. For 1:3 resonance, we find the anomalously strong and nonmonotonic dependence of the decay rate of the low frequency mode on its amplitude, if the decay rate of the high-frequency mode is comparatively large. In this case the low-frequency mode driven close to resonance can have several branches of steady-state vibrations with constant amplitude. If the decay rates of the both modes are small compared to their coupling and internal nonlinearity, the dynamics corresponds to slowly decaying strongly nonsinusoidal oscillations of the vibration amplitude. Weak driving can make these vibrations stable.