Neural network Hessian eigenvalues explained by approximate symmetries

Researchers have developed a new explanation for the numerous near-zero eigenvalues of the Hessian matrix in neural networks. They propose that these vanishing eigenvalues stem from approximate symmetries within the network's parameterization, which they term weakly lifted pseudo-Goldstone modes. In deep linear networks, these symmetries are exact, leading to flat directions and zero modes. The introduction of a rectifier nonlinearity, like ReLU, perturbs these symmetries, causing them to break weakly. The study demonstrates this mechanism in various network architectures, including a two-layer student-teacher model and a network trained on CIFAR-10, suggesting the findings extend beyond fully connected layers to convolutional networks. AI

IMPACT Provides a theoretical framework for understanding the loss landscape geometry in neural networks, potentially aiding in optimization and model design.

RANK_REASON The cluster contains an academic paper detailing a new theoretical finding in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• CIFAR-10
• Hessian
• Neural Networks

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →