Phase-locked neural activity in the brainstem generating the human frequency following response (FFR) has been shown to preserve neural information about certain spectrotemporal attributes of complex sounds that likely contributes to the perception of speech, language, music, and segregation of concurrent sound streams. Here we review some evidence in support of the view that the temporal pattern of neural activity in the FFR does preserve information relevant to the representation of spectra and pitch of complex sounds. Specifically, FFR spectra for both steady state and time variant complex sounds show clear peaks at formant related harmonics that follow the changes in formant frequencies for time-variant sounds. Similarly, neural information relevant to steady- and time-variant pitch, pitch shifts, and changes in pitch salience are well preserved in the FFR and exhibits a strong correlation with behavioral measures. For inharmonic, frequency shifted, and unresolved harmonics stimuli that produce pitch shifts or multiple pitches, neural activity relevant to these perceived changes in pitch is primarily contained in the neural representation of the temporal fine structure (TFS). Both reverberation and noise degrade the neural representation of envelope and TFS with phase locking to ENV showing greater resilience. Overall, these results clearly suggest that the FFR provides a robust physiologic window to evaluate the nature of neural representation of spectra and pitch of complex sounds in normal and impaired ears, age related changes in neural encoding, and to understand the role of experience in shaping subcortical processing and its application to re-training and perceptual learning.