Gated DeltaNet
PulseAugur coverage of Gated DeltaNet — every cluster mentioning Gated DeltaNet across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
-
Alibaba's Qwen3.8-27B integrates vision and language, rivals larger models
Alibaba's Qwen team has released Qwen3.8-27B, a new open-weight model that integrates vision and language capabilities. This model boasts a large context window of 262,144 tokens, extensible to 1 million, and features f…
-
Alibaba releases open-weight Qwen3.8-Max with 2.4T parameters
Alibaba has released Qwen3.8-2.4T-A95B, marking the first open-weight release of a model in its Qwen-Max class. This new model boasts 2.4 trillion total parameters, with 95 billion active parameters per forward pass, ut…
-
New Modular TTT Framework Simplifies Test-Time Training Design
Researchers have introduced Modular TTT, a new framework designed to simplify the creation and analysis of test-time training (TTT) methods. This framework represents the inner learning process as a directed acyclic gra…
-
New TTCD framework enhances long-context language modeling during inference
Researchers have introduced Test-Time Context Distillation (TTCD), a novel framework for long-context language modeling that optimizes parameter updates during inference. Unlike previous methods, TTCD incorporates a sel…
-
Guide to understanding Moonshot AI's Kimi K3 model architecture
A Reddit post outlines a recommended reading order for understanding the Kimi K3 model by Moonshot AI. The suggested sequence begins with foundational papers on linear transformers and gated delta mechanisms, progressin…
-
Kimi Delta Attention Explained: From Quadratic to Linear Variants
This article delves into the Kimi Delta Attention (KDA) mechanism, a sophisticated variant of linear attention. It traces the evolution from quadratic attention to KDA, explaining how KDA addresses the limitations of ea…
-
Kernelized Linear Attention Boosts Model Capacity and Throughput
Researchers have developed Kernelized Linear Attention Activations (KATA), a new framework designed to overcome the capacity limitations of linear attention models. KATA utilizes symmetric cones and rank-one positive se…
-
New research explores unified routing for adaptive LLM efficiency · 2 sources tracked
Two new research papers explore methods to optimize the efficiency of large language models by dynamically adjusting computational resources based on token complexity. The first paper, "Linear Attention Architectures," …
-
Sparse Delta Memory boosts linear RNNs for better long-context recall
Researchers have introduced Sparse Delta Memory (SDM), a novel architecture designed to enhance the long-context recall capabilities of linear RNNs. By employing a sparse addressing scheme, SDM significantly increases t…
-
Gated DeltaNet advances efficient VLM memory with delta rule and forget gate
Researchers have detailed the Gated DeltaNet, a novel approach to efficient vision-language models (VLMs) that utilizes a linear-time recurrence with a fixed-size memory matrix. This method, developed by Yang et al. fro…
-
Olmo Hybrid language model shows improved scaling and expressivity
Researchers have introduced Olmo Hybrid, a new 7-billion parameter language model that combines recurrence and attention mechanisms. This hybrid architecture, featuring Gated DeltaNet layers, demonstrates superior perfo…
-
New Taylor-Calibrate method improves Transformer to linear attention model conversion
Researchers have developed Taylor-Calibrate, a new initialization method designed to improve the conversion of Transformer models into hybrid linear attention models. This technique addresses the brittleness of converti…
-
xLSTM outperforms Mamba-2 and DeltaNet in sequence modeling tasks
A new research paper compares three subquadratic architectures—xLSTM, Mamba-2, and Gated DeltaNet—for sequence modeling tasks. The study found that xLSTM outperformed the others in code-model pre-training, distillation,…
-
Dynamic convolutions boost Transformer performance in LLMs
Researchers have introduced dynamic short convolutions as a new primitive to enhance Transformer architectures used in large language models. These dynamic convolutions utilize input-dependent filters, increasing expres…
-
New framework unifies sequence models using Bayesian memory
Researchers have introduced a "design-model" framework for creating efficient recurrent sequence maps based on memory assumptions. This framework uses Bayesian filtering to write evidence into memory and a query-depende…
-
New Chess Benchmark Exposes Transformer State-Tracking Failures
Researchers have introduced Chess-World-Model, a new benchmark designed to evaluate the state-tracking capabilities of world models. This benchmark utilizes a dataset of 10 million chess games to test a model's ability …
-
New Oryx Model Flexibly Switches Between Attention and Recurrent Mixers
Researchers have introduced Oryx, a novel hybrid model designed to flexibly switch between different sequence mixers, such as quadratic attention and linear recurrences, throughout a given sequence. This approach allows…
-
PapersWithCode adds multi-metric leaderboards and external paper support
Hugging Face has launched new features for PapersWithCode, a platform tracking AI state-of-the-art. The updates include support for multiple metrics on leaderboards, such as for Automatic Speech Recognition and Object D…
-
WriteSAE enables direct manipulation of recurrent language model states
Researchers have developed WriteSAE, a novel sparse autoencoder designed to manipulate the matrix updates within recurrent language model states. This method learns rank-1 matrix atoms that directly replace the model's …
-
NVIDIA unveils Gated DeltaNet-2 for improved linear attention
NVIDIA has introduced Gated DeltaNet-2, a new linear attention layer designed to improve memory editing in recurrent neural networks. This model separates the processes of erasing old information and writing new informa…