New theory explains grokking in deep neural networks via L2 phase transitions

Researchers have developed a new theory explaining the phenomenon of "grokking" in deep neural networks, where a model abruptly begins to generalize after a period of overfitting. The study, published on arXiv, proposes that grokking is related to hysteresis in first-order L2 phase transitions. By deliberately trapping models in metastable states, the researchers demonstrated that SGD noise can drive them across energy barriers, leading to escape times that follow Arrhenius scaling, thus reproducing the grokking curve. AI

IMPACT Provides a theoretical framework for understanding and potentially improving generalization in deep learning models.

RANK_REASON Academic paper published on arXiv detailing a new theoretical explanation for a phenomenon in deep learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Arrhenius scaling
• arXiv
• Deep Neural Networks
• grokking
• Hugging Face
• Ibrahim Talha Ersoy
• L2 regularization
• SGD

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →