Objectives: Sarcopenia is an age-associated skeletal muscle disorder that can profoundly impact the health of elderly people. However, the efficacy of herbal medicine in sarcopenia is uncertain. This review aims to investigate evidence of the effect of herbal medicine on sarcopenia. Methods: We systematically searched 12 electronic databases for relevant randomized controlled trials (RCTs). Only trials that met the inclusion criteria were selected, and the characteristics of the included studies were extracted and synthesized in a narrative manner. The quality of the included studies was assessed using Cochrane’s Risk of Bias (RoB) 2.0 tool. Results: 7 RCTs involving 672 participants with sarcopenia met the inclusion criteria. The intervention combining herbal medicine and conventional treatment (i.e., exercise, nutritional support) had a significant therapeutic effect compared with the conventional treatment, showing improvement in muscle strength (i.e., grip strength), muscle mass (i.e., appendicular skeletal muscle mass index), and physical function (i.e., gait speed, short physical performance battery, and timed up and go test). However, the methodological quality of the included RCTs was relatively low due to their high RoB, making it difficult to evaluate the efficacy of herbal medicine in sarcopenia. In terms of safety, several adverse events were reported. Conclusion: This review suggests that herbal medicine has a positive effect on muscle strength, muscle mass, and physical performance in elderly patients with sarcopenia, but there is a clear need for further research in this area.