Alcohol consumption can have significant deleterious consequences, including brain atrophy, neuronal loss, poorer white matter fiber integrity, and cognitive decline, but the effects of light-to-moderate alcohol consumption on brain structure remain unclear. Here we examine the associations between alcohol intake and brain structure using structural, diffusion tensor, and neurite orientation dispersion and density imaging data from 19,825 generally healthy middle-aged and older adults from the UK Biobank. Systematically controlling for potential confounds, we found that greater alcohol consumption was associated with lower global gray and white matter volume, regional gray matter volume in cortical and subcortical areas, and white matter fiber integrity and complexity. Post hoc analyses revealed that these associations were non-linear. Our findings extensively characterize the associations between alcohol intake and gray and white matter macrostructure and microstructure. Consuming two or more units of alcohol per day, equivalent to one drink in some establishments, could have negative effects on brain health, an important public health finding. One Sentence Summary Moderate alcohol intake, consuming two or more units of alcohol per day, has negative effects on brain health.