Both normal aging and neurodegenerative disorders such as Alzheimer's disease (AD) cause morphological changes of the brain. It is generally difficult to distinguish these two causes of morphological change by visual inspection of magnetic resonance (MR) images. To facilitate making this distinction and thus aid the diagnosis of neurodegenerative disorders, we propose a method for developing a spatio-temporal model of morphological differences in the brain due to normal aging. The method utilizes groupwise image registration to characterize morphological variation across brain scans of people with different ages. To extract the deformations that are due to normal aging we use partial least squares regression, which yields modes of deformations highly correlated with age, and corresponding scores for each input subject. Subsequently, we determine a distribution of morphologies as a function of age by fitting smooth percentile curves to these scores. This distribution is used as a reference to which a person's morphology score can be compared. We validate our method on two different datasets, using images from both cognitively normal subjects and patients with Alzheimer disease (AD). Results show that the proposed framework extracts the expected atrophy patterns. Moreover, the morphology scores of cognitively normal subjects are on average lower than the scores of AD subjects, indicating that morphology differences between AD subjects and healthy subjects can be partly explained by accelerated aging. With our methods we are able to assess accelerated brain aging on both population and individual level. A spatio-temporal aging brain model derived from 988 T1-weighted MR brain scans from a large population imaging study (age range 45.9–91.7y, mean age 68.3y) is made publicly available at www.agingbrain.nl.