Estimation of Over-Parameterized Models from an Auto-Modeling Perspective
Journal of the American Statistical Association · 2025 · Volume 120 · Issue 552
DOI: 10.1080/01621459.2025.2455192
Abstract
From a model-building perspective, we propose a paradigm shift for fitting over-parameterized models. Philosophically, the mindset is to fit models to future observations rather than to the observed sample. Technically, given an imputation method to generate future observations, we fit over-parameterized models to these future observations by optimizing an approximation of the desired expected loss function based on its sample counterpart and an adaptive duality function . The required imputation method is also developed using the same estimation technique with an adaptive m -out-of- n bootstrap approach. We illustrate its applications with the many-normal-means problem, n < p linear regression, and neural network-based image classification of MNIST digits. The numerical results demonstrate its superior performance across these diverse applications. While primarily expository, the article conducts an in-depth investigation into the theoretical aspects of the topic. It concludes with remarks on some open problems. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.
Keywords
Bootstrap, Cross–validation, Future observations, Image classification, Resampling
Enable JavaScript for citation tools and additional paper interactions.