Camera-based remote pulse rate monitoring can be used during fitness exercise to optimize the effectiveness of a workout. However, such monitoring suffers from vigorous body motions and dynamic illumination changes due to exercise, which may lead to erroneous estimates. To better cope with this, we propose a quality metric, comprised of a front-end metric and a back-end metric, to indicate the monitoring conditions (e.g. luminance, skin property) and assess the reliability of pulse rate measurement (e.g. signal quality). The proposed quality metric has been thoroughly benchmarked on 78 videos recorded in a fitness setting. The experimental results show that (i) appropriate light source intensity variation and its angle variation in the front-end metric are critical indicators for pulse rate measurement accuracy, and (ii) the back-end metric can effectively indicate/reject unreliable estimates. The proposed method in this paper is the first quality metric for camera-based pulse rate monitoring, validated for the challenging use-case of fitness exercises.