Background: Coffee is one of the most widely consumed beverages worldwide and has been of considerable interest in research on cognition and dementia. Objective: To investigate the effect of coffee on preclinical brain MRI markers of dementia and cognitive performance. Methods: In 2,914 participants from the population-based Rotterdam Study (mean age: 59.3±7.2 years, 55 females), we assessed coffee consumption, performed brain MRI, and assessed cognition at baseline. To study cognitive change, cognitive assessment was repeated after 5 years of follow-up. Coffee consumption was analyzed continuously (per cup increase) and in categories (0-1,>1-3,>3 cups/day). Using logistic and linear regression, associations of coffee consumption with lacunar infarcts and brain tissue volumes on MRI, and cognitive performance (cross-sectional and longitudinal) were investigated, adjusting for relevant confounders. Results: We found that higher coffee consumption was associated with a lower prevalence of lacunar infarcts [odds ratio per cup increase: 0.88 (95 CI:0.79;0.98)], and smaller hippocampal volume [difference: -0.01 (95 CI:-0.02;0.00)]. Also, we found that the highest category of coffee consumption was associated with better performance on the Letter Digit Substitution Task [difference: 1.13(95 CI:0.39;1.88)], Word Fluency test [0.74(95 CI:0.04,1.45)], Stroop interference task [1.82(95 CI:0.23;3.41)], and worse performance on the 15-Word Learning test delayed recall [-0.38(95 CI:-0.74;-0.02)]. These associations were not found when cognition was analyzed longitudinally. Conclusion: We found complex associations between coffee consumption, brain structure, and cognition. Higher coffee consumption was cross-sectionally associated with a lower occurrence of lacunar infarcts and better executive function, but also with smaller hippocampal volume and worse memory function.