Deep Learning Spectra Explained by Unifying Theory

Researchers have developed an analytic explanation for the low-dimensional eigenspectra observed in deep learning matrices. This phenomenon, previously explained by empirical observations or partial theoretical models, is now unified under the concept of Unconstrained Feature Models (UFMs). The study demonstrates that Deep Neural Collapse (DNC) is the underlying cause, with eigenvalues and eigenvectors derivable from feature means. The findings apply to both linear and ReLU networks and have been validated numerically across various architectures and datasets. AI

IMPACT Provides a unifying theoretical framework for understanding spectral properties in deep learning models.

RANK_REASON Academic paper detailing a new theoretical framework for understanding deep learning phenomena. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Connall Garrod
• Deep Learning
• Deep Neural Collapse
• ReLU networks
• Unconstrained Feature Models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →