Calibrating Prediction Regions

Abstract

Suppose that the variable X to be predicted and the learning sample Y n that was observed have a joint distribution, which depends on an unknown parameter θ. The parameter θ can be finite- or infinite-dimensional. A prediction region D n for X is a random set, depending on Y n , that contains X with prescribed probability α . This article studies methods for controlling simultaneously the conditional coverage proability of D n , given Y n , and the overall (unconditional) coverage probability of D n . The basic construction yields a prediction region D n , which has the following properties in regular models: Both the conditional and overall coverage probabilities of D n converge to α as the size n of the learning sample increases. The convergence of the former is in probability. Moreover, the asymptotic distribution of the conditional coverage probability about α is typically normal; and the overall coverage probability tends to α at rate n −1 . Can one reduce the dispersion of the conditional coverage probability about α and increase the rate at which overall coverage probability converges to α ? Both issues are addressed. The article establishes a lower bound for the asymptotic dispersion of conditional coverage probability. The article also shows how to calibrate D n so as to make its overall coverage probability converge to α at the faster rate n −2 . This calibration adjustment does not affect the asymptotic distribution or dispersion of the conditional coverage probability, in a first-order analysis. In general, a bootstrap Monte Carlo algorithm accomplishes the calibration of D n . In special cases, analytical calibration is possible.