Optically Pumped Magnetometer (OPM) can be utilized to realize a flexible and wearable Magnetoencephalography (MEG) system that improves the tracking accuracy and detected signal-to-noise ratio (SNR). However, the OPM-MEG is highly sensitive to environmental magnetic-field interference and thus noise reduction is critical for high-quality signal detection. In this work, we demonstrated a virtual gradiometer-based noise reduction method for wearable OPM-MEG, where three reference OPM sensors are configured as the basis set to map the environmental interference. The reference sensors were placed on the support and the subject's scalp respectively and the head model, alpha-rhythm and auditory evoked response experiments were performed. The results show that the proposed method can remarkably reduce the common-mode magnetic-field noise in the whole detection frequency range. Even though the applied white magnetic-field noise reaches up to 1 pT/Hz1/2, the noise-floor of the denoised data can still be maintained near the limited sensitivity of the OPM sensors. Meanwhile, high SNR alpha-rhythm and auditory evoked response signals for the actual subjects can be measured under the noise correction by the proposed method.