Conservative hypothesis tests and confidence intervals using importance sampling

Abstract

Importance sampling is a common technique for Monte Carlo approximation, including that of p -values. Here it is shown that a simple correction of the usual importance sampling p -values provides valid p -values, meaning that a hypothesis test created by rejecting the null hypothesis when the p -value is at most α will also have a Type I error rate of at most α . This correction uses the importance weight of the original observation, which gives valuable diagnostic information under the null hypothesis. Using the corrected p -values can be crucial for multiple testing and also in problems where evaluating the accuracy of importance sampling approximations is difficult. Inverting the corrected p -values provides a useful way to create Monte Carlo confidence intervals that maintain the nominal significance level and use only a single Monte Carlo sample.