In the unknown marine environment, due to special circumstances, the velocity measurement error of Doppler velocity log will increase, which will reduce the navigation accuracy of a SINS/GNSS/DVL integrated navigation system. To tackle this problem, an adaptive Kalman filter algorithm using Q-learning reinforcement learning is proposed. By taking the integrated navigation system as the environment, and the reciprocal of current position error as the reward, the proposed algorithm adaptively obtains the optimal velocity measurement noise estimation. Simulation results show that the proposed algorithm can effectively overcome the influence of the external current velocity, reduce the velocity error and position error, and improve the integrated navigation performance.