Research calls attention to the divergent school and labor market trajectories of Europe’s youth while, across the Atlantic, researchers describe the long-lasting consequences of poverty on adolescent development. In this paper we incorporate both processes to shed a new light on a classic concern in the sociology of stratification: how are adolescents’ aspirations, expectations, and school performance shaped by the combined socioeconomic contexts of family, school and neighborhood life? Theoretically, social contexts provide children with cultural resources that may foster their ambitions and bolster their academic performance. Reference group theory instead highlights how seemingly positive settings can depress educational performance as well as aspirations and expectations. We empirically test these competing claims, drawing on the Avon Longitudinal Study of Parents and Children (ALSPAC) which describes the school and neighborhood trajectories of 7,934 British children followed from birth to adolescence. We find that, generally, childhood school and neighborhood deprivation is negatively associated with adolescents’ school performance, aspirations and expectations for their future, in line with the cultural resource perspective. However, there are important exceptions to this pattern which point to reference group processes for (1) children of highly-educated parents, whose academic performance especially suffers from growing up in a poor neighborhood, and (2) for children from low-educated parents, whose academic aspirations and expectations are unexpectedly high when they either went to an affluent school or lived in an affluent neighborhood—but not both. We conclude by discussing implications for theory, policy and future research.