For autonomous driving of construction machinery such as construction crane, it is necessary to detect objects (e.g. person) in a construction site using the camera hung from the crane’s boom. Since the camera is attached to the boom, it is in motion as the boom moves from one place to another. In this research, we propose a camera motion compensation and person detection technique in a construction site for moving camera. The motion parameters are estimated using point-to-point features correspondences with RANSAC-like algorithm. The algorithm selects the best model with few randomly selected sample points. After compensating the motion effect, a Bayesian interface along with the Yolo model (called Yollo-Bayes model) is used for person detection. Two types of Bayes models are proposed to detect missing person by the Yolo. Bayes model for moving person is used to detect the person when he/she is in motion and Yolo fail to detect him/her. On the other hand, Bayes model for nonmoving person is used when the person is not in motion and Yolo miss to detect him/her. The motion estimation accuracy is confirmed by creating ground truth images with translation and rotation changes. To evaluate the performance of the Yolo-Bayes model, extensive experimental evaluations have been done with a lot of video images collected from the real construction site. It reveals that the proposed Yolo-Bayes model outperforms Yolo model. The proposed model is more effective when the number of training sample is small (F β – score is 5.77% higher than Yolo) or the model is tested on a completely new dataset (F β – score is 13.75% higher than Yolo).