Abstract: To match closely on a propensity, prognostic or other index score is harder than it would appear. Nearest-neighbor matching may closely align subjects on an estimate of the score, particularly when permitted to exclude subjects without nearby counterparts. Yet even after matching within strict tolerances, on a well-specified and efficiently estimated index, paired differences on actual scores can be surprisingly large. A first contribution of this research is to marshall evidence that univariate propensity score matching typically does leave large discrepancies on an underlying true score, regardless of how closely one matches on estimated scores. Even the luxury (a) of well-behaved (sub-gaussian) covariates that are (b) of moderate dimension (d=o(√n)) does not in itself address the problem. Nor does (c) strict overlap between intervention and control groups (all propensities within [a,b] ⊆ (0,1)), or matching in observance of caliper restrictions as suggested in extant literature. These conclusions are supported by insights from high-dimensional probability, and illustrated with health policy examples.

There's good news as well. When (a) and (b) are satisfied, estimation errors of the scores tend uniformly to zero, having sizes that are themselves estimable and can be summarized with a single number. That number in turn suggests a new caliper criterion that rejects few or no pairings that are close on the underlying score, yet is restrictive enough to cause index differences within the matched sample to tend uniformly to 0. None of (a)-(c) is required for this, although all will be true of the matched sample if the matching procedure has been governed by a caliper of the proposed type. The presentation will focus on propensity scores, but similar considerations apply to prognostic scores, principal scores and predictive mean matching.