Safe testing
Peter Grünwald, Rianne de Heide, Wouter Koolen
Journal of the Royal Statistical Society: Series B (Statistical Methodology) · 2024 · Volume 86 · Issue 5
Abstract
We develop the theory of hypothesis testing based on the e -value, a notion of evidence that, unlike the p -value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e -values are safe, i.e. they preserve type-I error guarantees, under such optional continuation . We define growth rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e -variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e -values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a 1-sample safe t -test and the 2 × 2 contingency table. Sharing Fisherian, Neymanian, and Jeffreys–Bayesian interpretations, e -values may provide a methodology acceptable to adherents of all three schools.
Keywords
Bayes factors, e-values, hypothesis testing, information projection, optional stopping, test martingales
Enable JavaScript for citation tools and additional paper interactions.