PulseAugur
LIVE 04:16:17
research · [3 sources] ·
0
research

New theory tracks spectral dynamics in wide neural networks

By PulseAugur Editorial · Summary by gemini-2.5-flash-lite from 3 sources

Researchers have developed a two-level dynamical mean-field theory to analyze the spectral dynamics within wide neural networks during training. This framework tracks both bulk and outlier spectral behaviors, offering insights into feature learning and how networks handle large output tasks like image classification and language modeling. The study suggests that while outlier dynamics are crucial for simpler tasks, complex tasks necessitate a restructuring of the spectral bulk for effective learning. AI

Summary written by gemini-2.5-flash-lite from 3 sources. How we write summaries →

IMPACT Provides a theoretical framework for understanding and potentially improving the training of large-scale neural networks.

RANK_REASON The cluster contains an academic paper detailing a new theoretical framework for analyzing neural network training dynamics.

Read on arXiv cs.AI →

COVERAGE [3]

  1. arXiv cs.LG TIER_1 · Florent Krzakala ·

    Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

    Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature …

  2. arXiv cs.AI TIER_1 · Blake Bordelon ·

    Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

    We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent. We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles whose spike directions remain st…

  3. arXiv stat.ML TIER_1 · Clarissa Lauditi, Cengiz Pehlevan, Blake Bordelon ·

    Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

    arXiv:2605.07870v1 Announce Type: cross Abstract: We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent. We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for…

RELATED ENTITIES

RELATED TOPICS