Image-based flame detection technology is an important research direction in the field of forest fire detection. Flame is a dynamically changing, luminescent chemical process, RGB images can only reflect two-dimensional information about the flames, which is not sufficient for distinguishing flames in any scene. With the development of infrared technology, the thermal properties of flames can be effectively visualized, and the use of IR features can well complement the characteristic information of the flame, which helps to improve the accuracy of flame detection. Therefore, we design a flame detection method based on adaptive feature fusion module, which aims to accomplish more accurate prediction using the two-dimensional features and the infrared features of the flame. The feature fusion module combines the channel attention mechanism and spatial attention mechanism of the image, which can selectively combine the RGB and infrared features of the flame and suppress the invalid feature information in the image. Experiments have demonstrated that the method can stably identify flames in forest scenes. In addition, we find that the method proposed in this paper also has good robustness for detecting obscured flames and small flames.