Abstract
Though stratified randomization achieves more balance on baseline covariates than pure randomization, it does affect the way we conduct inference. This paper considers the classical two-sample goodness-of-fit testing problem in randomized controlled trials when the researcher employs a particular type of stratified randomization—covariate-adaptive randomization. When testing the null hypothesis of equality of distributions between experimental groups in this setup, we first show that stratification leaves a mark on the test statistic's limit distribution, making it difficult, if not impossible, to obtain critical values. We instead propose an alternative approach to conducting inference based on a permutation test that i) is asymptotically exact in the sense that the limiting rejection probability under the null hypothesis equals the nominal α level, ii) is applicable under relatively weak assumptions commonly satisfied in practice, and iii) works for randomization schemes that are popular among empirically oriented researchers, such as stratified permuted block randomization.
The proposed test's main idea is that by transforming the original statistic by one minus its bootstrap p-value, it becomes asymptotically uniformly distributed on [0,1]. Thus, the transformed test statistic—also called prepivoted—has a fixed limit distribution that is free of unknown parameters, effectively removing the effect of stratification. Consequently, a permutation test based on the prepivoted statistic produces a test whose limiting rejection probability equals the nominal level. We present further numerical evidence of the proposed test's advantages in a Monte Carlo exercise, showing our permutation test outperforms the existing alternatives. We illustrate our method's empirical relevance by revisiting a field experiment by Butler and Broockman (2011) on the effect of race on state legislators' responsiveness to help their constituents register to vote during elections in the United States. Lastly, we provide the companion R Package package to facilitate and encourage applying our test in empirical research.
Local Organizers: Vincenzo Scrutinio, Annalisa Loviglio