In order to smooth the multi-scale texture with strong gradient while maintaining the weak structure which is challenging for the existing texture filtering methods, we put forward a novel algorithm based on multi-scale inherent variation features. Based on statistical analysis of various kinds of structure/texture pixels, six-dimensional discriminating features are found and first extracted from the multi-scale inherent variation curve, which demonstrate superiority in recognizing the structure pixels. Then, a preliminary structure prediction map can be obtained with a structure/texture classification model, which is generated by the cross-validation-based SVM training process. Next, we design a post-processing-based fine structure detection scheme to deal with the defects in the structure prediction map with three main steps successively, i.e., removing the mistaken texture pixels with thinning and outlier rejection, retrieving the missed structure pixels with breakpoints connection, and repositioning the structure pixels with structure correction. Finally, we propose a structure guided adaptive image smoothing method to smooth texture while preserving structure without halo effect. Experimental results show that our algorithm works better than the state-of-the-art methods in the preservation of weak structure as well as the suppression of texture with strong gradient or varying scales. [ABSTRACT FROM AUTHOR]