VideoSAR realizes continuously scene observation and expands SAR's range-azimuth two-dimensional scattering information to range-azimuth-time three-dimensional scattering information. Considering the effects of speckle noise, a novel speckle filter for VideoSAR image was proposed in this paper using the three-dimensional information of VideoSAR image. The proposed method mainly contains three steps. Firstly, a context covariance matrix that contains spatial and temporal context information is constructed for each pixel. Then, the similar samples are selected for each pixel according to the similarity test of covariance matrices. Finally, a sample averaging estimator is applied based on the similar samples. Experiments on real VideoSAR data show that the proposed algorithm achieves effective speckle filtering performance while well preserving the edge texture.