Grokking explained by two training clocks theory

Researchers have developed a theoretical framework to explain the phenomenon of "grokking" in machine learning, where a model fits training data and learns a generalizable rule at different rates. They propose the concept of "two training clocks" to distinguish the rapid decrease in classification loss from the slower simplification of the model's internal representation. This theory, initially demonstrated with deep linear networks, is then extended to explain similar behavior in ReLU MLPs, suggesting a two-stage learning process where the classifier adapts first, followed by representation refinement. AI

IMPACT Provides a theoretical explanation for a key aspect of model learning, potentially guiding future model development and training strategies.

RANK_REASON The cluster contains an academic paper detailing a new theoretical framework for understanding a machine learning phenomenon. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Deep linear networks
• Grokking
• ReLU MLPs

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →