Recently, the deep learning neural networks (DNNs) based filters have demonstrated their advantages to remove artifacts or improve the performance in the area of image/video coding. However, the existing DNNs-based filters can handle intra coding distortion or quality enhancement approaches, thus not suitable for inter frames without the Rate Distortion Optimization (RDO) strategy in encoder side, which is not desirable for the implementation on hardware since all the existing filters need to be executed to make the optimal choice for the current content in encoder side. Also the RDO algorithm can only make the local optimal choice, which does not represent the global optimum, because of the long-term dependency between the inter frames. In this paper, we propose an in-loop filter for inter frames to completely replace all the conventional filters in the codec, including de-blocking filter (DB), bilateral filter (BF), adaptive loop filter (ALF) and SAO (Sample Adaptive Offset) without the RDO strategy in encoder side. Moreover, the greedy heuristic is adopted to produce the optimal filter weights which approximates the global optimal solution in each stage during the off-line training. In the experiments, our method reduces the average BD-rate by 2.77%, 7.01%, 8.64% for luma and both chroma components with Random Access (RA) configuration by using only one set of parameters to handle multiple distortion and decrease the consumption of memory.