Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
Researchers have analyzed scaling laws and spectral properties of shallow neural networks operating within the feature learning regime. Their work, leveraging connections to compressed sensing and LASSO, details a phase diagram for excess risk exponents based on sample complexity and weight decay. This analysis reveals distinct scaling regimes and plateau behaviors that align with empirical observations in deep learning, and establishes a theoretical link between the spectral properties of network weights and generalization performance. AI
IMPACT Provides theoretical grounding for empirical observations in deep learning, potentially informing future model development.