Researchers have analyzed the dynamics of stochastic gradient descent (SGD) in two-layer neural networks trained on a multi-index teacher model. They decomposed the first-layer weights into a signal component aligned with the teacher and an orthogonal complement, observing that SGD dynamics induce teacher-dependent spectral spikes in the complement Gram matrix. This emergent anisotropy amplifies representation variance and degrades generalization, a phenomenon they term Decomposed Dynamics. Simulations with ReLU networks showed that freezing the orthogonal complement directions improved generalization, highlighting a mechanism by which learning features outside the teacher subspace can harm performance. AI
IMPACT Provides a theoretical explanation for how feature learning outside the primary teacher subspace can degrade generalization in neural networks.
RANK_REASON The cluster contains an academic paper detailing theoretical research on neural network training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →