Small infrared moving target detection has an important role in the sea-based infrared search and tracking, maritime area surveillance and other applications. This method aims to detect the small infrared moving targets with the sea-sky background. The present study proposes a detection algorithm for small infrared targets based on the spatio-temporal saliency fusion. The contourlet analysis and edge extraction are carried out in the concurrent design. In order to effectively suppress the background and improve the target signal-clutter ratio, the spectral residual method is combined with the abovementioned methods to reconstruct the target fusion saliency image. Then the target motion region is estimated based on the optical flow method for the fusion saliency image and it is matched with the target area of interest to achieve the moving target detection. Moreover, the pipeline filtering is introduced to achieve the target confirmation by multi-frame judgment, reduce false alarm rate and complete the moving target detection for the infrared image sequence. Experimental results show that the proposed algorithm can achieve continuous target detection and have a higher detection precision via real long wave infrared image sequences.