Neural Network Grokking Tied to Weight Norm Dynamics

Researchers have investigated the phenomenon of "grokking" in neural networks, where generalization occurs significantly after the model has already fit the training data. Their study suggests that the weight norm plays a crucial role in this delayed generalization. By intervening and manipulating the weight norm during training, they found that a specific critical norm value, Wc, is consistently reached, and this value scales with the network's modular base as a power law. Furthermore, they observed that holding the norm at a fixed multiple of Wc results in a grokking delay that follows an exponential relationship with the norm multiple. AI

RANK_REASON This is a research paper detailing a new finding about neural network behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• grokking
• LayerNorm
• Neural Networks
• Rho
• SARS-CoV-2 Alpha variant
• T_grok
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Wellington College

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →