In this paper, we propose an autonomous driving robot using binarized neural network (BNN) with Random Forest (RF). The proposed robot implements all functions from camera shooting to motor control on a small scale Field Programmable Gate Array (FPGA). For taken pictures, BNN with RF decides the direction of the proposed robot to left, right, and straight. In the experiments, we evaluate the circuit area, execution time, power consumption, and energy consumption of the proposed robot. Due to the use of BNN with RF, we can implement all functions into a small scale FPGA. Also, the proposed robot consumes about 61% less energy compared to an autonomous driving robot based on Raspberry Pi.