Sensitivity Analysis for Equivalence and Difference in an Observational Study of Neonatal Intensive Care Units

Abstract

In randomized experiments, it is sometimes important to demonstrate that two treatments do not differ greatly in their effects, or to demonstrate that the treatments have very different effects on one outcome but similar effects on another outcome. These “demonstrations” may take the form of rejecting a null hypothesis of inequivalence in an equivalence test, or rejecting a null hypothesis of equal and inequivalent effects on two outcomes. The procedures often express a complex hypothesis in terms of component hypotheses, and combine the component significance levels to test the complex hypothesis. If used in a randomized trial, randomization provides valid significance levels for each component test, and hence also for the combined procedure. In an observational study—that is, in a study of treatments that were not randomly assigned—there is typically concern that significance levels for testing hypotheses about treatment effects may be distorted by failure to control for some unobserved pretreatment covariate. This concern is raised in the evaluation of virtually all observational studies. The possible impact of such an unobserved covariate is clarified and displayed by a sensitivity analysis that, for various possible magnitudes of potential bias, yields a corresponding interval of possible significance levels. Some observational studies are sensitive to very small unobserved biases, whereas others are insensitive to large biases. Here, existing sensitivity analyses for component hypotheses are used to generate sensitivity analyses for complex hypotheses tested by combining component significance levels. We apply the procedure to our study of the timing of discharges of premature babies from neonatal intensive care units, focusing on the possible impact of delayed discharge on use of unplanned medical care after discharge.