The polarized hyperspectral images (PHSI) include the multidimensional information of polarization, spectral, spatial and radiant features, which provide more information about objects and background than traditional spectrum or intensity ones. However, the conventional object detection algorithms of PHSI are based on the Stokes vector, which mainly take advantage of the spectral information, and often ignore the continuous variation characteristics of polarized dimension, being similar to the spectrum. What is more, when the data is large, it will increase the calculative difficulty and error. Hence, a fourth-order tensor matched filtering (FTMF) is proposed in this paper, which is applied to the original data directly without extracting the Stokes vector, and achieves the combined utilization of polarization and spectrum. The experimental results also show that the proposed method in this paper is more suitable for the object detection of the PHSI.