The World Heart Federation (WHF) criteria, published in 2012, provided an evidence-based guideline for the minimal diagnosis of echocardiographically-detected RHD. Primary aim of the study was to determine whether use of the WHF criteria altered the threshold for the diagnosis of echocardiographically-detected RHD compared with the previous WHO/NIH criteria. A secondary aim was to explore the utility of a three reviewer reporting system compared to a single or two reviewer reporting structure. 144 de-identified echocardiograms (RHD, congenital valvar abnormality, physiological valvar regurgitation) were independently reported using the WHF criteria by two reviewers blinded to the previous WHO/NIH diagnosis. If there was discordance between the two reviewers, a third cardiologist independently performed a tie-breaker review. There was a 21% reduction of cases classified as RHD using the WHF criteria compared to the modified WHO/NIH criteria (68 cases compared to 86, p = 0.04). There was a 60% consensus across the different diagnostic categories with 2 reviewers, 89% majority agreement with 3 reviewers. 11% required an open label discussion. There was moderate agreement between 2 reviewers for any RHD, kappa 0.57 (CI 0.44–0.70), with no significant difference in agreement between the different categories. The WHF criteria have raised the threshold for the diagnosis of RHD compared to the WHO/NIH criteria. However, inter-reporter variability of the WHF criteria is high. A three reviewer system is likely more accurate than a single or two reporter system for the diagnosis of mild RHD. This has resource implications for echocardiographic screening programmes. • The WHF criteria raise the threshold of diagnosis of echocardiographically-detected RHD compared to WHO/NIH criteria. • This study reveals that there is only moderate rather than high inter-observer agreement of mild RHD changes. • The study recommends a panel of 3 reviewers for evaluation of abnormal cases detected by RHD screening programs, but this has significant resource implications for many regions. [ABSTRACT FROM AUTHOR]