Quantum cryptography has a strong security that classical cryptography cannot match, and quantum key distribution (QKD) is the most important method of quantum cryptography. Reference-frame-independent measurement-device-independent QKD (RFI-MDI-QKD) protocol can avoid the drift of the reference frame and all side channels of measurement devices. However, RFI MDI-QKD usually employs a weak coherent state (WCS) as the light source, which limits the transmission distance of secure keys. Here, by using heralded single-photon sources, we investigate an improved decoy-state RFI-MDI-QKD scheme. We perform numerical simulations for the scheme under different scenarios, such as considering the finite size and different drift angles. Compared with the scheme with WCS, our scheme can significantly improve the transmission distance of RFI-MDI-QKD.