linear attention
PulseAugur coverage of linear attention — every cluster mentioning linear attention across labs, papers, and developer communities, ranked by signal.
5 day(s) with sentiment data
-
SpectralShift enhances Gated DeltaNet context windows via spectral reparameterization · 2 sources tracked
Researchers have introduced SpectralShift, a novel method for extending the context window of Gated DeltaNet (GDN) models, which utilize linear attention mechanisms. Unlike previous approaches that focused on continued …
-
Kalman Delta Networks enhance language models with uncertainty-aware memory
Researchers have introduced Kalman Delta Networks (KDNs), a new family of models designed to enhance associative memory in language models by incorporating uncertainty awareness. These networks reformulate recurrent ass…
-
New Hybrid Transformer Architecture Improves Long-Context Extrapolation
Researchers have developed a new framework called Head-wise Hybrid Architecture (HwH) that re-evaluates the design of modern Transformers. By analyzing head-level functional organization using metrics like RoPE Frequenc…
-
Video world models struggle to track hidden states, study finds
Researchers have investigated the limitations of video world models in tracking unobserved states, even when they exhibit high visual fidelity. Using a "Shell Game" task, they found that models like Bidirectional and au…
-
New arXiv papers survey depth estimation progress and introduce novel diffusion model
Two new arXiv papers explore advancements in monocular depth estimation, a fundamental computer vision task. The first paper provides a comprehensive survey of the field, tracing its evolution from early methods to the …
-
Sliding Window Attention Outperforms Linear Attention in LLMs
A new research paper indicates that sliding window attention (SWA) with attention sinks performs as well as or better than linear attention models for large language models (LLMs). The study, published on arXiv and high…
-
Z.ai releases GLM-5.3-Flash, a multimodal MoE model with 1M context
Z.ai has launched GLM-5.3-Flash, a natively multimodal mixture-of-experts model with 320 billion total parameters and 18 billion active parameters per token. This model boasts a 1 million token context window and suppor…
-
New method adapts Vision Transformers for faster object detection
Researchers have developed a new method called Detector-Interface Distillation (DiD) to adapt Vision Transformers (ViTs) from Softmax attention to linear attention for object detection tasks. This label-free approach fo…
-
New QED method enhances long-range recall in linear attention models
A new research paper introduces Query-derived Erase Direction (QED), a method to improve long-range recall in linear attention models. QED adds a second erase direction derived from the query, orthogonal to the key, whi…
-
KV Cache Emerges as LLM Bottleneck, Driving Attention Variant Innovations
The KV cache, a critical component in autoregressive decoding for LLMs, is identified as the primary bottleneck for frontier models in 2026. Its size grows linearly with context length and batch size, making it the domi…
-
Panda framework enables real-time unsupervised anomaly detection in pelvic MRI
Researchers have developed a novel unsupervised anomaly detection framework called Panda, designed for real-time pelvic MRI imaging. This system utilizes a frozen DINOv3 Vision Transformer encoder and a noisy MLP bottle…
-
Paper explains why relative positional encodings improve transformer generalization
A new paper proposes an optimization-based explanation for why transformers with relative positional encodings generalize better to longer sequences than those with absolute encodings. The research suggests that the imp…
-
New methods tackle LLM long-context efficiency challenges · 3 sources tracked
Researchers are developing new methods to improve the efficiency of long-context reasoning in large language models. One approach, LISA, combines linear attention with a sparse attention mechanism to reduce computationa…
-
New method uses sparse embeddings for better cold item recommendation
Researchers have developed a new method for improving cold item recommendation in large-scale digital platforms by utilizing sparse embeddings instead of traditional dense vectors. This approach addresses the challenge …
-
HOLA enhances linear attention models with a complementary memory system
Researchers have developed a novel approach called HOLA (Hippocampal Linear Attention) to enhance the memory capabilities of linear attention and state-space language models. This method introduces a complementary 'hipp…
-
Sequential correlations impact in-context learning in sequence models
A new research paper explores how sequential correlations in data affect in-context learning (ICL) within modern sequence models. The study, using a solvable model based on linear attention and tested on transformer arc…
-
New HOLA architecture enhances linear attention language models with dual memory system
Researchers have developed HOLA (Hippocampal Linear Attention), a novel architecture that enhances linear attention language models by incorporating a complementary memory system. This system addresses the issue of info…
-
New Key-Correlated Layer Attention offers linear complexity for neural networks
Researchers have developed Key-Correlated Layer Attention (KCLA), a novel mechanism designed to improve how different layers within a neural network interact. KCLA addresses the quadratic computational complexity of tra…
-
Flexformer introduces learnable attention kernels for efficient Transformers
Researchers have introduced Flexformer, a novel linear Transformer architecture designed to overcome the quadratic complexity limitations of traditional Transformers. Flexformer achieves this by learning attention kerne…
-
New attention mechanisms boost LLM efficiency and reduce hallucination · 10 sources tracked
Researchers are developing novel attention mechanisms to improve the efficiency and capabilities of large language models (LLMs) and multimodal large language models (MLLMs). These advancements focus on optimizing spars…