A geometrical analysis of kernel ridge regression and its applications
Georgios Gavrilopoulos, Guillaume Lecué, Zong Shang
The Annals of Statistics · 2025 · Volume 53 · Issue 6
Abstract
We obtain upper bounds for the estimation error of Kernel Ridge Regression (KRR) for all nonnegative regularization parameters, offering a geometric perspective on various phenomena in KRR. As applications: 1. We address the Multiple Descents problem, unifying the proofs of (In Proceedings of Thirty Third Conference on Learning Theory (2020) 2683–2711 PMLR) and (Ann. Statist. 49 (2021) 1029–1054) for polynomial kernels in nonasymptotic regime and we establish Multiple Descents for the generalization error of KRR for polynomial kernel under sub-Gaussian design in asymptotic regimes. 2. In the nonasymptotic regime, we have established a one-sided isomorphic version of the Gaussian Equivalence Conjecture for sub-Gaussian design vectors. 3. We offer a novel perspective on the linearization of kernel matrices of nonlinear kernels, extending it to the power regime for polynomial kernels. 4. Our theory extends the results in (J. Mach. Learn. Res. 24 (2023) Paper No. [123]) under weak moment assumption. Our proof is based on three mathematical tools developed in this paper that can be of independent interest: 1. Dvoretzky–Milman theorem for ellipsoids under (very) weak moment assumptions. 2. Restricted isomorphic property in reproducing kernel Hilbert spaces with embedding index conditions. 3. A concentration inequality for finite-degree polynomial kernel functions.
Keywords
62F35, benign overfitting, deep learning models, Dvoretsky–Milman theorem, kernel ridge regression, restricted isomorphy property
Enable JavaScript for citation tools and additional paper interactions.