Objective: To provide a comprehensive understanding of APOE e4 effects across the lifespan on the 3 main neuroimaging biomarkers. Methods: Two hundred seven community-dwelling, cognitively normal APOE e4 carriers and noncarriers aged 20–87 years were involved in this study. They underwent structural MRI, fluorodeoxyglucose-PET, and florbetapir-PET scans. The effects of APOE , age, and APOE × age interaction were assessed voxel-wise for each modality. Results: There was no significant effect of APOE or APOE × age interaction on gray matter volume and glucose metabolism, although decreases with age tended to be stronger in noncarriers than in carriers. In contrast, β-amyloid (Aβ) deposition was significantly higher in carriers compared with noncarriers in a largely distributed network, and there was a significant APOE × age interaction such that Aβ deposition increased nonlinearly with age in APOE e4 carriers only. Conclusions: Our findings highlight a differential effect of APOE e4 on amyloid vs neurodegeneration biomarkers. APOE e4 mainly influences Aβ deposition, while the effects on gray matter volume and glucose metabolism are at best subtle. ClinicalTrials.gov identifier: NCT01638949.