Gated DeltaNet
PulseAugur coverage of Gated DeltaNet — every cluster mentioning Gated DeltaNet across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
Olmo Hybrid language model shows improved scaling and expressivity
Researchers have introduced Olmo Hybrid, a new 7-billion parameter language model that combines recurrence and attention mechanisms. This hybrid architecture, featuring Gated DeltaNet layers, demonstrates superior perfo…
-
New Taylor-Calibrate method improves Transformer to linear attention model conversion
Researchers have developed Taylor-Calibrate, a new initialization method designed to improve the conversion of Transformer models into hybrid linear attention models. This technique addresses the brittleness of converti…
-
xLSTM outperforms Mamba-2 and DeltaNet in sequence modeling tasks
A new research paper compares three subquadratic architectures—xLSTM, Mamba-2, and Gated DeltaNet—for sequence modeling tasks. The study found that xLSTM outperformed the others in code-model pre-training, distillation,…
-
Dynamic convolutions boost Transformer performance in LLMs
Researchers have introduced dynamic short convolutions as a new primitive to enhance Transformer architectures used in large language models. These dynamic convolutions utilize input-dependent filters, increasing expres…
-
New framework unifies sequence models using Bayesian memory
Researchers have introduced a "design-model" framework for creating efficient recurrent sequence maps based on memory assumptions. This framework uses Bayesian filtering to write evidence into memory and a query-depende…
-
New Chess Benchmark Exposes Transformer State-Tracking Failures
Researchers have introduced Chess-World-Model, a new benchmark designed to evaluate the state-tracking capabilities of world models. This benchmark utilizes a dataset of 10 million chess games to test a model's ability …
-
New Oryx Model Flexibly Switches Between Attention and Recurrent Mixers
Researchers have introduced Oryx, a novel hybrid model designed to flexibly switch between different sequence mixers, such as quadratic attention and linear recurrences, throughout a given sequence. This approach allows…
-
PapersWithCode adds multi-metric leaderboards and external paper support
Hugging Face has launched new features for PapersWithCode, a platform tracking AI state-of-the-art. The updates include support for multiple metrics on leaderboards, such as for Automatic Speech Recognition and Object D…
-
WriteSAE enables direct manipulation of recurrent language model states
Researchers have developed WriteSAE, a novel sparse autoencoder designed to manipulate the matrix updates within recurrent language model states. This method learns rank-1 matrix atoms that directly replace the model's …
-
NVIDIA unveils Gated DeltaNet-2 for improved linear attention
NVIDIA has introduced Gated DeltaNet-2, a new linear attention layer designed to improve memory editing in recurrent neural networks. This model separates the processes of erasing old information and writing new informa…