Despite numerous studies during the last decade speech emotion recognition is still the task of limited success. Great efforts were made for extending emotional speech feature sets and selecting the most effective ones, proposing multi-stage and multiple classifier based classification schemes, and developing multi-modal speech emotion recognition technique. Nevertheless, the reported emotion recognition rates vary from 70 % up to 90 % depending on the analyzed language, the number of recognized emotions, the speaker mode, and other important factors. Considering the nonlinear and fluctuating nature of the spoken language, we present a feature set, based on a fractal dimension (FD) for emotion classification. Katz, Castiglioni, Higuchi, and Hurst exponent-based FD features were employed in 2–7 emotion classification tasks. The experimental results show a clear superiority of FD based feature sets against acoustic ones. The feature selection enabled us to reduce the initial feature set down to 2–7 order sets and to improve thereby the accuracy of speech emotion classification by 11.4 %. The obtained average classification accuracy for all tasks was 96.6 %.