We investigated the association between hearing loss and cognitive impairment in the National Institute for Longevity Sciences, Longitudinal Study for Aging. We used the data from the 2nd to 7th study waves, which covered the years 2000–2012. On average, the prevalence of hearing loss was 44.8%, and the prevalence of hearing aid usage was 1.25% in the mild hearing loss group, 22.4% in the moderate hearing loss group, and 77.4% in the severe to profound hearing loss group. We recruited 1193 participants with hearing loss (3260 cumulative observations) for the generalized estimating equation analysis. The dependent variable was cognitive impairment; the independent variables were the pure-tone average threshold levels at 4 frequencies (500, 1000, 2000, and 4000 Hz) by the better hearing ear (PTABHE) and hearing aid usage; and the covariates were age, sex, depressed mood, smoking status, alcohol intake, education, obesity, physical activity, household income, and history of hypertension, dyslipidemia, diabetes, ischemic heart disease, stroke, ear disease, and occupational noise exposure. PTABHE was significantly associated with cognitive impairment in both the mild and moderate hearing loss groups. In the moderate hearing loss group, the odds ratio of cognitive impairment was significantly lower among regular users of hearing aids than among non-users.