The quantum Fisher information (QFI) in SU(2) and SU(1,1) interferometers was considered, and the QFI-only calculation was overestimated. In general, the phase estimation as a two-parameter estimation problem, and the quantum Fisher information matrix (QFIM) is necessary. In this paper, we theoretically generalize the model developed by Escher et al [Nature Physics 7, 406 (2011)] to the QFIM case with noise and study the ultimate precision limits of SU(2) and SU(1,1) interferometers with photon losses because photon losses as a very usual noise may happen to the phase measurement process. Using coherent state and squeezed vacuum state as a specific example, we numerically analyze the variation of the overestimated QFI with the loss coefficient, and find its disappearance and recovery phenomenon.