Positive feedbacks driving habitat-forming species recovery and population growth are often lost as ecosystems degrade. For such systems, identifying mechanisms that limit the re-establishment of critical positive feedbacks is key to facilitating recovery. Theory predicts the primary drivers limiting system recovery shift from biological to physical as abiotic stress increases, but recent work has demonstrated that this seldom happens. We combined field and laboratory experiments to identify variation in limitations to coral recovery along an environmental stress gradient at Ningaloo Reef and Exmouth Gulf in northwest Australia. Many reefs in the region are coral depauperate due to recent cyclones and thermal stress. In general, recovery trajectories are prolonged due to limited coral recruitment. Consistent with theory, clearer water reefs under low thermal stress appear limited by biological interactions: competition with turf algae caused high mortality of newly settled corals and upright macroalgal stands drove mortality in transplanted juvenile corals. Laboratory experiments showed a positive relationship between crustose coralline algae cover and coral settlement, but only in the absence of sedimentation. Contrary to expectation, coral recovery does not appear limited by the survival or growth of recruits on turbid reefs under higher thermal stress, but to exceptionally low larval supply. Laboratory experiments showed that larval survival and settlement are unaffected by seawater quality across the study region. Rather, connectivity models predicted that many of the more turbid reefs in the Gulf are predominantly self seeded, receiving limited supply under degraded reef states. Overall, we find that the influence of oceanography can overwhelm the influences of physical and biological interactions on recovery potential at locations where environmental stressors are high, whereas populations in relatively benign physical conditions are predominantly structured by local ecological drivers. Such context-dependent information can help guide expectations and assist managers in optimizing strategies for spatial conservation planning for system recovery.