Brief

NVIDIA has introduced Gated DeltaNet-2, a new linear attention layer designed to improve memory editing in recurrent neural networks. This model separates the processes of erasing old information and writing new information using distinct channel-wise gates, addressing a limitation in previous delta-rule architectures. Trained on 100 billion tokens with 1.3 billion parameters, Gated DeltaNet-2 demonstrates superior performance over existing models like Mamba-2 and KDA, particularly in long-context retrieval tasks. AI

Researchers are developing new attention mechanisms to handle increasingly long contexts in large language models. One approach, Runtime-Certified Bounded-Error Quantized Attention, uses tiered KV caches to compress memory while guaranteeing fallback to exact attention, ensuring quality for tasks like language modeling and retrieval. Another method, DashAttention, employs differentiable sparse hierarchical attention to adaptively select relevant tokens, achieving high sparsity with comparable accuracy to full attention and offering improved performance over existing hierarchical methods. Variational Linear Attention (VLA) reframes linear attention as a regularized least-squares problem, limiting state norm growth and improving associative recall accuracy, while also achieving significant speedups. AI