What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy
Researchers have investigated the phenomenon of 'grokking' in neural networks, where a model transitions from memorization to generalization. Their findings indicate that the weight norm, previously thought to be the primary driver of this transition, primarily acts as an upstream handle for the logit scale. By manipulating the logit scale directly, researchers could control the grokking delay across its entire range, with the weight norm contributing only a minor additional effect. This relationship was found to be dependent on the loss function used, with mean-squared error showing a different mechanism than cross-entropy. AI
IMPACT Clarifies the underlying mechanisms of generalization in neural networks, potentially informing future model architectures and training strategies.