Delta rule
PulseAugur coverage of Delta rule — every cluster mentioning Delta rule across labs, papers, and developer communities, ranked by signal.
-
New SANE technique stabilizes extreme-context recurrent models
Researchers have developed a new technique called State Anomaly Neutralization (SANE) to improve the stability of Delta-Rule recurrent models when processing extremely long contexts. These models, which typically have $…
-
Moonshot AI unveils Kimi K3, largest open-weight model with novel memory tech
Moonshot AI has developed Kimi K3, an open-weight model boasting 3 trillion parameters, making it the largest of its kind. The innovation lies not just in scale but in a novel memory management system called Kimi Delta …
-
Ant Group releases Ling-3.0-Flash, a 124B parameter hybrid AI model
Ant Group's AI division has released Ling-3.0-Flash, a new hybrid reasoning model with 124 billion parameters that activates 5.1 billion parameters per computation. This model aims to match or exceed the performance of …
-
New training method improves recurrent memory model performance
A new research paper introduces the "Orthogonalized Read," a training technique designed to improve the performance of recurrent memory models. This method, applied during the read phase of multiplicative LSTMs, acts as…
-
NVIDIA unveils Gated DeltaNet-2 for improved linear attention
NVIDIA has introduced Gated DeltaNet-2, a new linear attention layer designed to improve memory editing in recurrent neural networks. This model separates the processes of erasing old information and writing new informa…
-
OSDN improves linear attention with online preconditioning
Researchers have introduced OSDN, a novel method that enhances linear attention mechanisms by incorporating provable online preconditioning. This technique augments the Delta Rule with a diagonal preconditioner, which i…