Abstract: Competing approaches to inference in randomized experiments differ primarily in (1) which notion of ``no treatment effect’’ being tested; and (2) whether or not a superpopulation model is posited for the potential outcomes. Recommended hypothesis tests in a given paradigm may be invalid even asymptotically when applied in other frameworks, creating the risk of misinterpretation by practitioners when a given method is deployed. For a large class of test statistics common in practice, we develop a general framework for ensuring validity across competing modes of inference. To do this, we employ permutation tests based upon prepivoted test statistics, wherein a test statistic is first transformed by a suitably constructed cumulative distribution function and its permutation distribution is then enumerated. In essence, the approach uses the permutation distribution of a p-value for a large-sample test known to be valid under the null hypothesis of no average treatment effect as a reference distribution. The framework readily accommodates regression-adjusted estimators of average treatment effects, and the corresponding tests are never less powerful asymptotically than a test based upon the unadjusted estimator and maintain asymptotic validity even if the regression model is misspecified. The tests retain finite-sample exactness under stricter definitions of no treatment effect such as Fisher’s sharp null by virtue of being permutation tests, and validity across different superpopulation models can be ensured through the choice of the estimated CDF used when prepivoting.

http://www.mit.edu/~cfogarty/