Testing Multiple Forecasters
Econometrica · 2008 · Volume 76 · Issue 3
DOI: 10.1111/j.1468-0262.2008.00847.x
Abstract
We consider across-calibrationtest of predictions by multiple potential experts in a stochastic environment. This test checks whether each expert is calibrated conditional on the predictions made by other experts. We show that this test is good in the sense that a true expert—one informed of the true distribution of the process—is guaranteed to pass the test no matter what the other potential experts do, and false experts will fail the test on all but a small (category I) set of true distributions. Furthermore, even when there is no true expert present, a test similar to cross-calibration cannot be simultaneously manipulated by multiple false experts, but at the cost of failing some true experts.
Keywords
Calibration, probability forecasting, testing
Enable JavaScript for citation tools and additional paper interactions.