Thyroid hormone (TH) is crucial during neurodevelopment, but high levels of TH have been linked to neurodegenerative disorders. No data on the association of thyroid function with brain imaging in the general population are available. We therefore investigated the association of thyroid-stimulating hormone and free thyroxine (FT4) with magnetic resonance imaging (MRI)-derived total intracranial volume, brain tissue volumes, and diffusion tensor imaging measures of white matter microstructure in 4683 dementia- and stroke-free participants (mean age 60.2, range 45.6–89.9 years). Higher FT4 levels were associated with larger total intracranial volumes (β = 6.73 mL, 95% confidence interval = 2.94–9.80). Higher FT4 levels were also associated with larger total brain and white matter volumes in younger individuals, but with smaller total brain and white matter volume in older individuals (p-interaction 0.02). There was a similar interaction by age for the association of FT4 with mean diffusivity on diffusion tensor imaging (p-interaction 0.026). These results are in line with differential effects of TH during neurodevelopmental and neurodegenerative processes and can improve the understanding of the role of thyroid function in neurodegenerative disorders.