Background: The accumulation of amyloid-β1−42 (Aβ1−42) peptides and phosphorylated-Tau181 (p-Tau181) tangles from the preclinical stages of Alzheimer’s disease (AD) has led to a biological definition of the disease. However, among Aβ1−42-positive individuals, cognitive decline onset varies, and some never develop symptoms. Genetic influences on molecular pathways and their interactions with proteinopathy may underlie this heterogeneity. Leveraging data from a large sample of cognitively intact older adults in the European Prevention of Alzheimer Dementia (EPAD) cohort, we examined how AD-related pathophysiological changes (i.e., Aβ1−42 and p-Tau181), polygenic pathways and their interaction are associated with WM micro- and macrostructural properties. Methods: We selected 803 individuals (mean age = 64.7 ± 7.3 years, 458 [57.0%] females, 275 [34.2%] APOE-ε4 carriers) with CSF-Aβ1−42 and p-Tau181 measurements available, full genotyping, and structural and diffusion MRI. Polygenic risk scores (PRSs) were computed using 85 AD-related genetic variants. These were mapped to their corresponding genes and, after excluding those belonging to the APOE locus, clustered by function into six pathway-specific PRSs (i.e., immune activation, signal transduction, inflammation, lipid, amyloid, and clearance pathways). Diffusion MRIs were processed through the fixel-based analysis framework to derive fiber density (FD) and fiber cross-section (FC) metrics, which were averaged within WM tracts. Linear models assessed the effects of AD-related pathophysiological changes, global and pathway-specific PRSs, and their interactions on FD and FC at both the tract and fixel levels. Models were corrected for multiple comparisons. Results: P-Tau181 was primarily associated with greater FD. The lipid pathway was associated with greater FD and FC, with these effects predominantly occurring in the left hemisphere, consistent with evidence of hemispheric dominance. The clearance pathway moderated the effect of Aβ1−42 on FD, with a positive slope in A + compared to A- individuals. The immune activation pathway moderated the effect of p-Tau181 on FD, with a negative slope in T + compared to T- individuals. Conclusions: Pathway-specific genetic vulnerability to AD is associated with alterations in WM tracts both directly and by moderating the effects of AD-related pathophysiological changes. AD-associated genetic risk should be integrated into the AD diagnostic framework to enable targeted screening and intervention for future preclinical trials aimed at specific biological pathways.