Weather recognition is an important and fundamental technology and it deeply affects the daily lives of human beings. On the contrary, few works focus on the researches of weather recognition. In recent years, with the development of deep learning and their successful applications of computer vision, some researches based on weather recognition have been obtained some progresses. However, it still is a challenge task to correctly recognize the weather classes because of the varieties of weather phenomena. Attention mechanism is an important way to design the network architecture. In this paper, to improve the recognition accuracy, we firstly propose a model to utilize the channel and spatial attention which can mainly focus on both the "what" and "where" features of objects. Secondly, an adaptive fusion strategy is proposed to effectively represents the internal relationships between the two types of attentions. Thirdly, the proposed model is a lightweight model and can be seamlessly integrated into any CNN baseline network to improve its representational abilities. Lastly, comprehensive experiments on two different kinds of datasets for recognition demonstrate the correctness and effectiveness of the proposed model. These experiments also show the generalities of the proposed model.