Benchmarking the Unobserved

Abstract

Researchers often use observational data to estimate the causal effect of a treatment on an outcome. The central threat to such estimates is an unobserved confounder: a variable that affects both the treatment and the outcome but is not measured. An omitted confounder biases the estimated effect, and this bias does not shrink as the sample grows. Sensitivity analysis addresses this threat by asking how strong a hidden confounder would need to be to overturn a result. A widely used method for linear regression, introduced by Cinelli and Hazlett [6] and extended by Chernozhukov et al. [5] and implemented in the sensemakr software, reports an upper bound on the possible bias together with a small set of summary statistics. The bound is valid for whatever confounder strengths the analyst specifies; it cannot supply those strengths, which are unknown. In practice they are supplied by benchmarking, which compares the confounder to an observed covariate. This makes two claims at once: that the confounder is no stronger than the covariate, and that the covariate is an appropriate reference for it. The second claim cannot be checked from the data. We study the formal leave-one-out version of this procedure and ask a question its validity proof leaves open: when this assumption is false, does the reported bound still contain the true bias? We answer with Monte Carlo simulations in which the confounder is known, so that the bias and the bound can be compared directly. The bound covers the true bias until the confounder reaches roughly the strength of the covariate it is benchmarked against, and then fails sharply rather than gradually. The strength at which it fails depends on the covariate set in raw terms, so to locate it we derive a relative-strength coordinate that expresses the confounder’s strength in the bound’s own units. In these units the failure sits at a single point across structurally different covariate sets, the point at which the confounder overtakes the benchmark, and it shifts predictably when the worst-case assumption behind the bound is relaxed. None of the summary statistics warn of any of this: as the bound begins to fail, every one moves in the direction that appears more reassuring. We conclude that these summary statistics, read on their own, do not establish that an estimate is robust to unobserved confounding; they establish robustness only when the benchmarking assumption holds. We recommend that analysts either defend this assumption explicitly or set the confounder’s strengths directly from subject-matter knowledge, rather than treating the reported statistics as sufficient.