The automation of self-driving agricultural vehicles, particularly self-driving tractors, has made significant strides in the growth of the agri-tech business. This self-driving agricultural vehicles will greatly aid the improvement of the legacy agricultural system in terms of safety, fuel economy, and time management. Driving a self-driving tractor on an unpaved path is more challenging than driving it on a well-paved road with defined operational design domains and proper road infrastructure. To the best of our knowledge, though the significance of this challenging task is very clear, no comprehensive survey to address this challenge has been conducted yet. To overcome this problem, we have performed a detailed survey of techniques used in self-driving tractors for different purposes. In addition to the literature survey critique analysis of existing solutions has been performed and open questions have been raised. These open questions will help researchers in this field to perform state-of-the-art research.