Rumor propagation is becoming easier and leads to severe consequences for society in several minutes or hours due to the rapid development of social networks. Thus, detecting rumors in early time is necessary and urgent for the community. In recent studies, machine learning approaches are widely applied in detecting rumors based on various features extracted from content, user characteristics, and propagation structure. Some studies have shown that features extracted from users are more valuable for rumor early detection. Whereas existing studies utterly utilize various user features, and all of them are deemed as equally, which ignore the inter-feature and temporal difference. Therefore, in this paper, we analyze the effectiveness of six frequently used user features with several representative deep learning models to learn more about such difference. And we propose a novel annealing attention model based on the analysis. The proposed model learns feature-attention and temporal-attention with multi-layer perceptron and parameterized annealing function to capture the difference and enhance the original user features in rumor early detection. Experimental results on the real-world dataset demonstrate that the proposed model detects rumors with an accuracy of 93.6% on Weibo in 15 minutes, which outperforms the state-of-art methods.