High-Dimensional Analysis for Generalized Nonlinear Regression: From Asymptotics to Algorithm
DOI:
https://doi.org/10.1609/aaai.v38i12.29253Keywords:
ML: Deep Learning Theory, ML: Deep Learning AlgorithmsAbstract
Overparameterization often leads to benign overfitting, where deep neural networks can be trained to overfit the training data but still generalize well on unseen data. However, it lacks a generalized asymptotic framework for nonlinear regressions and connections to conventional complexity notions. In this paper, we propose a generalized high-dimensional analysis for nonlinear regression models, including various nonlinear feature mapping methods and subsampling. Specifically, we first provide an implicit regularization parameter and asymptotic equivalents related to a classical complexity notion, i.e., effective dimension. We then present a high-dimensional analysis for nonlinear ridge regression and extend it to ridgeless regression in the under-parameterized and over-parameterized regimes, respectively. We find that the limiting risks decrease with the effective dimension. Motivated by these theoretical findings, we propose an algorithm, namely RFRed, to improve generalization ability. Finally, we validate our theoretical findings and the proposed algorithm through several experiments.Downloads
Published
2024-03-24
How to Cite
Li, J., Liu, Y., & Wang, W. (2024). High-Dimensional Analysis for Generalized Nonlinear Regression: From Asymptotics to Algorithm. Proceedings of the AAAI Conference on Artificial Intelligence, 38(12), 13500-13508. https://doi.org/10.1609/aaai.v38i12.29253
Issue
Section
AAAI Technical Track on Machine Learning III