Weight norm's role in neural network grokking clarified

Researchers have investigated the phenomenon of 'grokking' in neural networks, where a model transitions from memorization to generalization. Their findings indicate that the weight norm, previously thought to be the primary driver of this transition, primarily acts as an upstream handle for the logit scale. By manipulating the logit scale directly, researchers could control the grokking delay across its entire range, with the weight norm contributing only a minor additional effect. This relationship was found to be dependent on the loss function used, with mean-squared error showing a different mechanism than cross-entropy. AI

IMPACT Clarifies the underlying mechanisms of generalization in neural networks, potentially informing future model architectures and training strategies.

RANK_REASON The item is an academic paper detailing research findings on a machine learning phenomenon. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• cross-entropy
• DagsHub
• Gotit.pub
• grokking
• Hugging Face
• IArxiv
• LayerNorm
• logit scale
• mean squared error
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →