Background and aims In a large stroke-free population, we sought to identify cardiovascular risk factors and carotid plaque components associated with carotid plaque burden, lumen volume and stenosis. Methods The carotid arteries of 1562 stroke-free participants from The Rotterdam Study were imaged on a 1.5-Tesla MRI scanner. Inner and outer wall of the carotid arteries were automatically segmented and lumen volume (mm³), wall volume (outer wall–inner wall) and plaque burden (wall volume/outer wall volume) (%) were quantified. Plaque components were visually determined and luminal stenosis was assessed. We analyzed associations of cardiovascular risk factors and carotid plaque components with plaque burden and lumen volumes using regression analysis. Results We investigated 2821 carotid plaques and found that women had larger plaque burden (50.7 ± 7.8% vs. 49.2 ± 7.7%, p <0.0001) and smaller lumen volumes (933 ± 286 mm³ vs. 1078 ± 334 mm³, p <0.0001) than men. In women, age, HDL-cholesterol and systolic blood pressure, and in men, total cholesterol, non-HDL cholesterol and statin use were independently associated with higher plaque burden and lumen volume. Furthermore, smoking and diabetes were associated with lumen volume in men (respectively p = 0.04 and p = 0.002). Intraplaque hemorrhage (IPH) and lipid were related to a larger plaque burden (OR 1.30 [1.05–1.60] and OR 1.28[1.06–1.55]). Finally, within the highest quartile of plaque burden, IPH was strongly associated with luminal stenosis independent of age, sex, plaque burden and composition (Beta = 15.2; [11.8–18.6]). Conclusions Several cardiovascular risk factors and plaque components, in particular IPH, are associated with higher plaque burden. Carotid IPH is strongly associated with an increased luminal stenosis.