Due to the presence of rain, the visibility of images captured outdoors on rainy days will be severely degraded. Rain removal using image processing technology can reduce the influence of rain to estimate rain-free images. However, existing traditional rain removal methods are very time-consuming, and newly emerging deep learning-based methods require a large amount of data and computational resources, resulting in long time consumption and poor visual effect. To solve the problems of existing methods, a new method is proposed to remove oblique rain streaks in windy conditions better. Firstly, the directional gradient constraint is proposed to locate oblique rain streaks in the rain layer effectively. Then, a sparse prior for oblique rain streaks is presented to enhance oblique rain streaks removal. After that, an optimization problem combining the directional gradient, sparse priors, and non-negativity constraints is presented. Finally, the alternating direction method of multipliers is exploited to solve the optimization problem effectively. Experiment results show that our method outperforms other methods in removing oblique rain streaks and requires less time.