New theory explains incremental learning in shallow neural networks

Researchers have developed a new theoretical framework for understanding incremental learning in shallow neural networks. This work focuses on polynomial-width two-layer networks trained on orthogonal multi-index targets under standard initialization. The findings indicate that incremental learning still occurs, with the loss decreasing sequentially based on the Hermite expansion of the target, and lower-order components being learned before higher-order ones. AI

IMPACT Provides theoretical insights into neural network training dynamics, potentially informing future model architectures.

RANK_REASON The item is an academic paper detailing theoretical advancements in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• Charles Hermite
• computer science
• DagsHub
• Gotit.pub
• gradient descent
• Gradient Flow
• Hugging Face
• IArxiv
• Influence Flower
• Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry
• machine learning
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →