Under investigation in this paper is a variable-coefficient AB (vcAB) system, which describes marginally unstable baroclinic wave packets in geophysical fluids and ultra-short pulses in nonlinear optics. The modulation instability analysis of solutions with variable coefficients in the presence of a small perturbation is studied. The modified Darboux transformation (mDT) of the vcAB system is constructed via a gauge transformation. The first-order non-autonomous rogue wave solutions of the vcAB system are presented based on the mDT. It is found that the wave amplitude of exhibits two types of structures, i.e. the double-peak structure appears if the plane-wave solution parameter is equal to zero, while selecting yields a single-peak one. Effects of the variable coefficients on the rogue waves are graphically discussed in detail. The periodic rogue wave and composite rogue wave are obtained with different inhomogeneous parameters. Additionally, the nonlinear tunneling of the rogue waves through a conventional hyperbolic nonlinear well and barrier are investigated. [ABSTRACT FROM AUTHOR]