We investigated the association of specific retinal sublayer thicknesses on optical coherence tomography (OCT) with brain magnetic resonance imaging (MRI) markers. We included 2124 persons (mean age 67.0 years; 56% women) from the Rotterdam Study who had gradable retinal OCT images and brain MRI scans. Thickness of retinal nerve fiber layer (RNFL), ganglion cell layer (GCL), and inner plexiform layer were measured on OCT images. Volumetric, microstructural, and focal markers of brain tissue were assessed on MRI. We found that thinner RNFL, GCL, and inner plexiform layer were associated with smaller gray-matter and white-matter volume. Furthermore, we found that thinner RNFL and GCL were associated with worse white-matter microstructure. No association was found between retinal sublayer thickness and white-matter lesion volumes, cerebral microbleeds, or lacunar infarcts. Markers of retinal neurodegeneration are associated with markers of cerebral atrophy, suggesting that retinal OCT may provide information on neurodegeneration in the brain.