grokking
PulseAugur coverage of grokking — every cluster mentioning grokking across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
New optimizer CvAdamW accelerates neural network grokking
Researchers have introduced CvAdamW, a novel variant of the AdamW optimizer designed to accelerate the "grokking" phenomenon in neural networks. Grokking, where a model generalizes after memorizing training data, is oft…
-
New model explains neural network grokking by focusing on representability
Researchers have developed a new model to understand grokking in neural networks, a phenomenon where generalization is delayed. This model, using holomorphic monomial activations on modular arithmetic tasks, demonstrate…
-
New research details factors for effective AI model training acceleration
Researchers have identified key factors that influence the effectiveness of representational priors in accelerating AI model training, particularly in the phenomenon of grokking. Their findings indicate that the alignme…
-
Research identifies key factors for effective representational priors in AI model generalization
A new research paper explores the factors that make representational priors effective in machine learning, particularly in the context of "grokking," where models transition from memorization to generalization. The stud…
-
New technique accelerates neural network generalization by 52x
Researchers have introduced Geometric Dimensionality Regularization (GeomDR), a novel technique to influence the phenomenon of grokking in neural networks. Grokking, where models initially memorize data before generaliz…
-
New audit tool reveals representation compression lags generalization in neural networks
A new audit tool has been developed to analyze the grokking phenomenon in neural networks, specifically examining how representations compress after generalization. The tool reveals that for modular arithmetic tasks, em…
-
New research reveals weight direction, not magnitude, carries transferable circuit identity in neural networks
Researchers have developed a novel method called cross-trajectory chimera interventions to investigate the portability of learned features in neural networks. By splitting weight vectors into magnitude and direction com…
-
Research links AI grokking delay to representational structure formation
Researchers have investigated the phenomenon of grokking, where a model generalizes long after its training data has been fully memorized. Through experiments with a one-layer transformer, they causally demonstrated tha…
-
Grokking in ML requires breaking data symmetry for generalization
Researchers have investigated the phenomenon of grokking in machine learning, where a model achieves high training accuracy but only generalizes to new data much later. Their study, using the Recursive Feature Machine (…
-
Research: Addressable memory crucial for AI edit propagation, not just learning
A new research paper explores how neural networks learn and retain information, distinguishing between 'grokking' and 'edit propagation'. The study found that repeated shared access, whether through loop recurrence or m…
-
Transformer grokking delay linked to decoder bottleneck, study finds
A new research paper explores the phenomenon of 'grokking' in transformers, where models abruptly generalize after a long delay during training on algorithmic tasks. The study suggests this delay stems from limited acce…
-
Weight norm's role in neural network grokking clarified
Researchers have investigated the phenomenon of 'grokking' in neural networks, where a model transitions from memorization to generalization. Their findings indicate that the weight norm, previously thought to be the pr…
-
New theory explains grokking in deep neural networks via L2 phase transitions
Researchers have developed a new theory explaining the phenomenon of "grokking" in deep neural networks, where a model abruptly begins to generalize after a period of overfitting. The study, published on arXiv, proposes…
-
Neural Network Grokking Tied to Weight Norm Dynamics
Researchers have investigated the phenomenon of "grokking" in neural networks, where generalization occurs significantly after the model has already fit the training data. Their study suggests that the weight norm plays…
-
New metric predicts transformer 'grokking' phenomenon
A new research paper introduces the Frequency Synchronization Degree (FSD), a metric designed to predict the phenomenon of 'grokking' in transformer models. Grokking is characterized by a sudden improvement in a model's…
-
New Metric Predicts Transformer 'Grokking' Phenomenon
A new research paper introduces the Frequency Synchronization Degree (FSD), a metric to measure the synchronization of Fourier circuits in Grokking Transformers. This metric consistently predicts grokking, the phenomeno…
-
Grokking explained by two training clocks theory
Researchers have developed a theoretical framework to explain the phenomenon of "grokking" in machine learning, where a model fits training data and learns a generalizable rule at different rates. They propose the conce…
-
New papers analyze neural network grokking via spectral geometry
Two new arXiv papers explore the phenomenon of 'grokking' in neural networks, where models generalize only after memorizing training data. One paper proposes 'Low-Rank Decay' (LRD) as a spectral regularizer to improve g…
-
Researchers explore grokking phenomenon in ridge regression
Three new research papers explore the concept of "grokking" in machine learning, specifically within the context of ridge regression. One paper presents a numerical procedure to find optimal regularization strength, dem…
-
New Framework Decodes Deep Learning Phenomena: Grokking and Double Descent
Researchers have developed a new framework to analyze and explain complex learning dynamics in deep neural networks, specifically focusing on phenomena like grokking and double descent. This framework decomposes learnin…