In wind power generation systems, gears are frequently faulty components, and their failure rate accounts for a very high proportion of wind turbine failures. Fault diagnosis of wind turbine gears is beneficial for timely maintenance and can effectively reduce economic losses. This article utilizes an improved parameter adaptive optimization variational mode decomposition (VMD) method to decompose gear vibration signals and construct a fault feature vector set. By comparing VMD and Empirical Mode Decomposition (EMD) for decomposing multiple fault vibration signals of gears, it has been proven that VMD has higher decomposition accuracy. The parameters of VMD were optimized using the improved grey wolf optimization algorithm (IGWO). Through comparative experiments, it was verified that the IGWO algorithm has better optimization efficiency. Finally, through an example, it was verified that the gear fault diagnosis model based on IGWO-VMD has good recognition performance for different wear levels of gears.