New research details hierarchical learning in two-layer neural networks

Researchers have theoretically analyzed the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model. The study, which jointly optimizes both layers with a perturbative parameter controlling their relative training speeds, proves that the constant and linear components of the hidden link function are recovered within predicted timescales. The paper also examines the learning of the quadratic component, demonstrating that previously learned components continue to influence the dynamics. This analysis utilizes quantitative approximation results for singularly perturbed flows near integral constraint manifolds, and phenomenologically observes singular behavior in the empirical measure of weights as the quadratic component is learned. AI

IMPACT Provides theoretical insights into the learning mechanisms of deep neural networks, potentially informing future model architectures and training strategies.

RANK_REASON Academic paper published on arXiv detailing theoretical analysis of neural network learning dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• berthier2024learning
• Berthierville
• cs.LG
• Hugging Face
• Montanari
• Singular perturbations and hierarchical learning in two-layer neural networks
• zhou

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →