linear attention
PulseAugur coverage of linear attention — every cluster mentioning linear attention across labs, papers, and developer communities, ranked by signal.
- developed by Softmax 95%
- used by Softmax 95%
- developed ruler 90%
- competes with sliding-window attention 80%
- instance of Full Attention 70%
- instance of alphaXiv 70%
- affiliated with Full Attention 70%
- affiliated with State Space Models 70%
- instance of State Space Models 70%
- used by Gated DeltaNet 70%
- used by ruler 70%
- used by alphaXiv 70%
7 day(s) with sentiment data
-
KVBuffer enhances linear attention serving for long-context AI models
Researchers have introduced KVBuffer, a novel serving mechanism designed to enhance the efficiency of linear attention models, particularly for long-context inference. This IO-aware approach buffers recent keys and valu…
-
New FlyWheel attention kernels boost TPU performance up to 4.14x
Researchers have developed FlyWheel, a new set of attention kernels optimized for Google's Tensor Processing Units (TPUs). These kernels address inefficiencies in current TPU implementations by reformulating computation…
-
New research questions state tracking robustness in recurrent AI models
A new paper from arXiv explores the limitations of state tracking in recurrent neural network architectures, arguing that error control dynamics are as crucial as theoretical expressivity. The research demonstrates that…
-
New FSKA framework learns nonstationary attention kernels for efficient ML
Researchers have introduced Flow-based Spectral Kernel Attention (FSKA), a novel framework designed to learn nonstationary attention kernels for improved self-attention mechanisms in machine learning. FSKA utilizes spec…
-
New geometric approach to transformer attention mechanisms unveiled
This paper introduces a novel geometric framework for understanding and developing attention mechanisms within transformer architectures. The authors formalize concepts like 'knowledge worlds' and 'attention pairings' t…
-
New Theory Explains Orthogonalization's Effectiveness in Neural Networks
Researchers have developed a new theoretical framework to explain why orthogonalization techniques, such as Muon, are effective in optimizing neural networks. The study reveals that the Hessian of a neural network, whic…
-
New LLoPS framework enhances linear attention in transformers
Researchers have introduced LLoPS (Learnable Low-Rank Polynomial Sketch), a new framework designed to improve linear attention mechanisms in transformers. This method aims to reduce approximation errors inherent in prev…
-
New research probes transformer attention dynamics and efficiency
Recent research explores the intricacies of transformer attention mechanisms, revealing new insights into their training dynamics and effectiveness. One study demonstrates that for shallow transformers processing unboun…
-
Linear attention can only learn linear components of nonlinear functions
A new research paper explores the capabilities of linear attention mechanisms within transformers, particularly in the context of in-context learning. The study establishes a nonlinearity-noise equivalence, demonstratin…
-
New research explores softmax attention capabilities in Transformers
A new paper explores the theoretical underpinnings of softmax attention, a key component in Transformer models. Researchers demonstrate that softmax attention can effectively handle both linear and nonlinear statistical…
-
New research details hybrid attention for LLMs, boosting long-context performance
A new research paper, "Mechanics of Long-Context Hybrid Models Part 1.1," explores the effectiveness of hybrid attention mechanisms in large language models (LLMs) for handling long contexts. The study identifies a "See…
-
New research enhances linear attention efficiency and performance
Researchers are developing new methods to improve the efficiency and performance of linear attention mechanisms in large language models. One approach, Switching Linear Attention (SwiLA), enhances representational capac…
-
New attention mechanisms boost long-context sequence modeling
Two new papers introduce novel approaches to enhance long-context sequence modeling in recurrent neural networks. The first paper, "SMat-Attention," proposes Structured Matrix Attention, which uses structured causal mas…
-
SpectralShift enhances Gated DeltaNet context windows via spectral reparameterization · 2 sources tracked
Researchers have introduced SpectralShift, a novel method for extending the context window of Gated DeltaNet (GDN) models, which utilize linear attention mechanisms. Unlike previous approaches that focused on continued …
-
Kalman Delta Networks enhance language models with uncertainty-aware memory
Researchers have introduced Kalman Delta Networks (KDNs), a new family of models designed to enhance associative memory in language models by incorporating uncertainty awareness. These networks reformulate recurrent ass…
-
New Hybrid Transformer Architecture Improves Long-Context Extrapolation
Researchers have developed a new framework called Head-wise Hybrid Architecture (HwH) that re-evaluates the design of modern Transformers. By analyzing head-level functional organization using metrics like RoPE Frequenc…
-
Video world models struggle to track hidden states, study finds
Researchers have investigated the limitations of video world models in tracking unobserved states, even when they exhibit high visual fidelity. Using a "Shell Game" task, they found that models like Bidirectional and au…
-
New arXiv papers survey depth estimation progress and introduce novel diffusion model
Two new arXiv papers explore advancements in monocular depth estimation, a fundamental computer vision task. The first paper provides a comprehensive survey of the field, tracing its evolution from early methods to the …
-
Sliding Window Attention Outperforms Linear Attention in LLMs
A new research paper indicates that sliding window attention (SWA) with attention sinks performs as well as or better than linear attention models for large language models (LLMs). The study, published on arXiv and high…
-
Z.ai releases GLM-5.3-Flash, a multimodal MoE model with 1M context
Z.ai has launched GLM-5.3-Flash, a natively multimodal mixture-of-experts model with 320 billion total parameters and 18 billion active parameters per token. This model boasts a 1 million token context window and suppor…