Power system models need regular calibration to reflect the current system status, identify and mitigate potential issues. However, traditional calibration approaches usually have local minima issues that require intervention from domain experts. In this study, we propose a reinforcement learning algorithm based on hierarchical parameter tuning (HPT-RL) for power system model calibration. The proposed HPT-RL is inspired by the natural hierarchy in parameter tuning where its higher level learns which parameter to tune and its lower level learns the tuning step size. We implement HPT-RL using the Deep Deterministic Policy Gradient (DDPG) architecture. We evaluate and compare the performance of HPT-RL, traditional DDPG, Bayesian Optimization, and Random Search for a generator model calibration on a four-bus system simulated with PSS®E. The performance of HPT-RL was evaluated based on 100 repeated experiments for two scenarios, namely low randomization and high randomization, based on where a simulated disturbance occurs on the bus system. The results show that HPT-RL outperforms the other three benchmark methods.