The motion of a self-balancing car is characterized by nonlinearity, uncertainty, and strong coupling, making its control a challenging research area. As the car is used over time, the mechanical structure's lubrication in the servo system deteriorates, highlighting the system's nonlinear characteristics due to friction. Consequently, the traditional PID control method proves inadequate. To tackle this issue, in this paper, the friction force is identified by genetic algorithm and compensated into the control quantity. To prevent the algorithm from converging into a local optimal solution, a mutation operation inspired by complement coding is devised to enhance the mutation process. Simulation results demonstrate that the improved genetic algorithm enhances identification accuracy, speeds up convergence, and greatly improves control effectiveness. These findings hold significant theoretical and practical implications for both the practical application and further research on self-balancing car.