New Mesoscopic Model Explains Neural Network Learning Dynamics

Researchers have developed a new mesoscopic description for understanding learning dynamics in extensive-width neural networks. This approach characterizes the distribution of overlaps, which are correlations between learned and target features. In the early stages of learning, a Gaussian description of overlap statistics simplifies the dynamics into collective variables that track the learning of linear and quadratic components of the target function. The study also identifies a quadratic shielding mechanism that accelerates feature specialization in extensive-rank networks by filtering gradient fluctuations. AI

IMPACT Provides a new statistical framework for understanding how large neural networks learn complex functions, potentially guiding future model development.

RANK_REASON The cluster contains a research paper detailing a new theoretical model for neural network learning dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• Mesoscopic learning dynamics of extensive-width neural networks
• stochastic gradient descent
• two-layer neural networks

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →