GRASP: a goodness-of-fit test for classification learning

Journal of the Royal Statistical Society: Series B (Statistical Methodology) · 2024 · Volume 86 · Issue 1

Abstract

Performance of classifiers is often measured in terms of average accuracy on test data. Despite being a standard measure, average accuracy fails in characterising the fit of the model to the underlying conditional law of labels given the features vector ( ⁠ Y ∣ X ⁠ ), e.g. due to model misspecification, over fitting, and high-dimensionality. In this paper, we consider the fundamental problem of assessing the goodness-of-fit for a general binary classifier. Our framework does not make any parametric assumption on the conditional law Y ∣ X and treats that as a black-box oracle model which can be accessed only through queries. We formulate the goodness-of-fit assessment problem as a tolerance hypothesis testing of the form H 0 : E [ D f ( B e r n ( η ( X ) ) ‖ B e r n ( η ^ ( X ) ) ) ] ≤ τ where D f represents an f -divergence function, and η ( x ) ⁠ , η ^ ( x ) ⁠ , respectively, denote the true and an estimate likelihood for a feature vector x admitting a positive label. We propose a novel test, called G oodness-of-fit with Ra ndomisation and S coring P rocedure (GRASP) for testing H 0 ⁠ , which works in finite sample settings, no matter the features (distribution-free). We also propose model-X GRASP designed for model-X settings where the joint distribution of the features vector is known. Model-X GRASP uses this distributional information to achieve better power. We evaluate the performance of our tests through extensive numerical experiments.

Keywords

classification, goodness-of-fit, hypothesis testing, model-X

Enable JavaScript for citation tools and additional paper interactions.