Objective: To investigate the association of brain volumes, white matter lesion (WML) volumes, and lacunes, with cognitive decline in a population-based cohort of nondemented persons. Methods: Within the Rotterdam Study, 3624 participants underwent brain magnetic resonance imaging. Cognition was evaluated at baseline (2005 to 2009) and at the follow-up visit (2011 to 2013). We used a test battery that tapped into domains of executive function, information processing speed, motor speed, and memory. The volumetric measures assessed were total brain volume, lobar (gray matter and white matter) volumes, and hippocampal volumes. We also studied the association of WML volumes and lacunes with cognitive decline using linear regression models. Results: Total brain volume was associated with decline in global cognition, information processing, and motor speed (P<0.001) in analyses controlled for demographic and vascular factors. Specifically, smaller frontal and parietal lobes were associated with decline in information processing and motor speed, and smaller temporal and parietal lobes were associated with decline in general cognition and motor speed (P<0.001 for all tests). Total WML volume was associated with decline in executive function. Lobar WML volume, hippocampal volume, and lacunes were not associated with cognitive decline. Conclusions: Lower brain volume is associated with subsequent cognitive decline. Although lower total brain volume was significantly associated with decline in global cognition, specific lobar volumes were associated with decline in certain cognitive domains.