grokking
PulseAugur coverage of grokking — every cluster mentioning grokking across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
Research probes neural network grokking, revealing transfer benefits and relapse risks
A new research paper explores the mechanisms behind "grokking," a phenomenon where neural networks rapidly generalize after a period of poor performance. The study, conducted on modular arithmetic tasks, found that tran…
-
New complexity controller accelerates AI model grokking
Researchers have developed a new differentiable complexity estimator, $K^{\mathrm{CDM}}_{\mathrm{s}F}$, to study the phenomenon of grokking in machine learning. This novel approach allows for the application of calculus…
-
Grokking transition in neural networks quantified, data complexity key
Researchers have quantified the transition from memorization to generalization in neural networks, a phenomenon known as grokking. They discovered a power-law scaling relation for the onset time of grokking, indicating …
-
Biologically inspired mechanisms boost AI model generalization
Researchers have explored biologically inspired mechanisms to improve the grokking phenomenon in multilayer perceptrons, where models transition from memorization to generalization. By incorporating features like input …
-
New research details 'canalization' phenomenon in neural network generalization
Researchers have identified a phenomenon called "canalization" in overparameterized neural networks, which describes how the selection of solutions that fit training data evolves during training. This process, observed …
-
Research paper questions grokking transition mechanism in neural networks
A new research paper explores the phenomenon of "grokking" in neural networks, where a model initially performs poorly but then rapidly improves its generalization ability after memorizing the training data. The study i…
-
New optimizer CvAdamW accelerates neural network grokking
Researchers have introduced CvAdamW, a novel variant of the AdamW optimizer designed to accelerate the "grokking" phenomenon in neural networks. Grokking, where a model generalizes after memorizing training data, is oft…
-
New model explains neural network grokking by focusing on representability
Researchers have developed a new model to understand grokking in neural networks, a phenomenon where generalization is delayed. This model, using holomorphic monomial activations on modular arithmetic tasks, demonstrate…
-
New research details factors for effective AI model training acceleration
Researchers have identified key factors that influence the effectiveness of representational priors in accelerating AI model training, particularly in the phenomenon of grokking. Their findings indicate that the alignme…
-
Research identifies key factors for effective representational priors in AI model generalization
A new research paper explores the factors that make representational priors effective in machine learning, particularly in the context of "grokking," where models transition from memorization to generalization. The stud…
-
New technique accelerates neural network generalization by 52x
Researchers have introduced Geometric Dimensionality Regularization (GeomDR), a novel technique to influence the phenomenon of grokking in neural networks. Grokking, where models initially memorize data before generaliz…
-
New audit tool reveals representation compression lags generalization in neural networks
A new audit tool has been developed to analyze the grokking phenomenon in neural networks, specifically examining how representations compress after generalization. The tool reveals that for modular arithmetic tasks, em…
-
New research reveals weight direction, not magnitude, carries transferable circuit identity in neural networks
Researchers have developed a novel method called cross-trajectory chimera interventions to investigate the portability of learned features in neural networks. By splitting weight vectors into magnitude and direction com…
-
Research links AI grokking delay to representational structure formation
Researchers have investigated the phenomenon of grokking, where a model generalizes long after its training data has been fully memorized. Through experiments with a one-layer transformer, they causally demonstrated tha…
-
Grokking in ML requires breaking data symmetry for generalization
Researchers have investigated the phenomenon of grokking in machine learning, where a model achieves high training accuracy but only generalizes to new data much later. Their study, using the Recursive Feature Machine (…
-
Research: Addressable memory crucial for AI edit propagation, not just learning
A new research paper explores how neural networks learn and retain information, distinguishing between 'grokking' and 'edit propagation'. The study found that repeated shared access, whether through loop recurrence or m…
-
Transformer grokking delay linked to decoder bottleneck, study finds
A new research paper explores the phenomenon of 'grokking' in transformers, where models abruptly generalize after a long delay during training on algorithmic tasks. The study suggests this delay stems from limited acce…
-
Weight norm's role in neural network grokking clarified
Researchers have investigated the phenomenon of 'grokking' in neural networks, where a model transitions from memorization to generalization. Their findings indicate that the weight norm, previously thought to be the pr…
-
New theory explains grokking in deep neural networks via L2 phase transitions
Researchers have developed a new theory explaining the phenomenon of "grokking" in deep neural networks, where a model abruptly begins to generalize after a period of overfitting. The study, published on arXiv, proposes…
-
Neural Network Grokking Tied to Weight Norm Dynamics
Researchers have investigated the phenomenon of "grokking" in neural networks, where generalization occurs significantly after the model has already fit the training data. Their study suggests that the weight norm plays…