Canonical Regularisation of Wide Feature-Learning Neural Networks
A new paper introduces a novel framework for understanding and generalizing regularization in wide neural networks. The research identifies that standard ridge regularization can distort the inductive bias of feature-learning networks, particularly impacting pre-trained models. To address this, the authors axiomatize a regime-agnostic canonical regularizer and derive a generalized ridge, proposing "arc ridge" as a practical, robust surrogate that connects early stopping to canonical regularization across learning regimes. The theory is validated through empirical studies in image processing and NLP. AI
IMPACT Introduces a new theoretical framework for understanding and improving neural network training, potentially impacting model performance and generalization.