Structural and functional alterations of the brain in persons genetically at-risk for Alzheimer's disease (AD) are crucial in unravelling AD development. Filippini et al. found that the default mode network (DMN) is already affected in young APOE ε4-carriers, with increased co-activation of the DMN during rest and increased hippocampal task activation. We aimed to replicate the early findings of Filippini et al, using the APOE gene, still the principal AD risk gene, and extended this with a polygenic risk score (PRS) analysis for AD, using the Human Connectome Project dataset (HCP). We included participants from the HCP S1200 dataset (age range: 22-36 years). We studied morphometric features, functional DMN co-activation and functional task activation of recollection performance. Permutation Analysis of Linear Models (PALM) was used to test for group differences between APOE ε4-carriers and non-carriers, and to test the association with PRS. PALM controls for biases induced by the family structure of the HCP sample. Results were family-wise error rate corrected at p < 0.05. Our primary analysis did not replicate the early findings of Filippini et al. (2009). However, compared with non-carriers, APOE ε4-carriers showed increased functional activation during the encoding of subsequently recollected items in areas related to facial recognition (p<0.05, t>756.11). This increased functional activation was also positively associated with PRS (APOE variants included) (p<0.05, t>647.55). Our results are supportive for none to limited genetic effects on brain structure and function in young adults. Taking the methodological considerations of replication studies into account, the true effect of APOE ε4-carriership is likely smaller than indicated in the Filippini paper. However, it still holds that we may not yet be able to detect already present measurable effects decades before a clinical expression of AD. Since the mechanistic pathway of AD is likely to encompass many different factors, further research should be focused on the interactions of genetic risk, biomarkers, aging and lifestyle factors over the life course. Sensitive functional neuroimaging as used here may help disentangling these complex interactions.