In this work, a gender-to-age hierarchical analysis structure is proposed rather than directly classifying speech clips into gender and age categories. A two-stage Support Vector Machine (SVM) classifier is adopted to identify a female and male, and then conduct an age classification. To realize the gender recognition, the mean of the fundamental frequency and the standard deviation of the fast Fourier transform from speech clips are employed. Additionally, a part of 16 extracted speech characteristic parameters are used to understand human ages according to their genders. Notably, human utterance characteristics are considered to determine adequate speech parameters to minimize feature ambiguities among females and males under different ages. The experimental results demonstrate that the proposed gender-to-age hierarchical recognition scheme can achieve 17.9% accuracy-rate improvement in average, as compared to the results from the conventional direct classification scheme.