Detectingp-Hacking
Graham Elliott, Nikolay Kudrin, Kaspar Wüthrich
Econometrica · 2022 · Volume 90 · Issue 2
Abstract
We theoretically analyze the problem of testing forp-hacking based on distributions ofp-values across multiple studies. We provide general results for when such distributions have testable restrictions (are non-increasing) under the null of nop-hacking. We find novel additional testable restrictions forp-values based ont-tests. Specifically, the shape of the power functions results in both complete monotonicity as well as bounds on the distribution ofp-values. These testable restrictions result in more powerful tests for the null hypothesis of nop-hacking. When there is also publication bias, our tests are joint tests forp-hacking and publication bias. A reanalysis of two prominent data sets shows the usefulness of our new tests.
Keywords
p-values, p-curve, complete monotonicity, publication bias
Enable JavaScript for citation tools and additional paper interactions.