Background subtraction (BS) has been a norm for moving object detection along a classical computer vision pipeline, especially when the labelled data is largely unavailable. It has been widely used for infrastructure-based sensing such as traffic surveillance with roadside cameras. Existing BS algorithms focus on detecting moving objects, while the temporal motionless objects are neglected. This leads to performance degradation in particular at signalized intersections where vehicles may stop to wait in red. In this paper, we propose a hierarchical adaptive BS method which can eliminate the cumulative errors for those temporally static objects based on images from roadside fish-eye cameras in real-time. The proposed method is validated in both the CARLA simulation and the real-world environment. The results show that our method outperforms ViBe and LOBSTER by about 45% and 39%, respectively, on recall, without compromising too much in precision.