This paper proposes an FPGA design and implementation of Local Bundle Adjustment (LBA) for Stereo Visual Odometry. In order to implement the LBA module on FPGA, binary Cauchy weight is designed. The LBA module is generated by Vivado HLS and implemented on the FPGA (XCZU3EG). Compared with the performance running on the FPGA and that on the Intel and ARM Processor, the LBA accelerator has achieved the highest speed which could speed up the process by 2.5 times than that on the Intel CPU, and 7.5 times than that on the ARM CPU. Based on improved LBA method on FPGA, the localization deviation is constrained within 10mm.