New attention mechanisms boost LLM efficiency and reduce hallucination · 10 sources tracked
ByPulseAugur Editorial·[49 sources]·
Researchers are developing novel attention mechanisms to improve the efficiency and capabilities of large language models (LLMs) and multimodal large language models (MLLMs). These advancements focus on optimizing sparse attention for long contexts, reducing computational costs, and mitigating issues like hallucination and weak visual grounding. Techniques such as Flash Sparse Attention (FSA), Information-Regularized Attention (IRA), and Multipole Semantic Attention (MuSe) aim to enhance performance, reduce latency, and enable LLMs to handle longer sequences more effectively. Additionally, methods like ADAPT and MATCH are being explored to improve the faithfulness and reasoning abilities of MLLMs by directly addressing attention dynamics and integrating in-context retrieval.
AI
IMPACT
These advancements aim to improve LLM efficiency, reduce hallucination, and enhance reasoning capabilities, potentially accelerating the adoption of long-context models.
RANK_REASON
Multiple research papers introducing new attention mechanisms and frameworks for LLMs and MLLMs.
arXiv:2607.12789v1 Announce Type: new Abstract: The $\mathcal{O}(N^2)$ complexity of attention over $N$ tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to $\mathcal{O}(MN)$ by representing keys with $M$ codewords, but …
arXiv:2607.11976v1 Announce Type: new Abstract: Indexer-TopK, the operation to compute the scores and select the top-k candidates, is widely used by sparse attention kernels in large language models and vector retrieval in recommendation systems and vector databases. However, exi…
The $\mathcal{O}(N^2)$ complexity of attention over $N$ tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to $\mathcal{O}(MN)$ by representing keys with $M$ codewords, but applies uniform codebook capacity regardless of …
arXiv:2607.09694v1 Announce Type: cross Abstract: Attention Residuals replace the fixed residual sum with depthwise attention over previous sub-layer outputs in large language models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with r…
arXiv cs.AI
TIER_1English(EN)·Victor J. B. Jung, Gagandeep Singh, Joseph Melber, Kristof Denolf, Francesco Conti, Luca Benini·
arXiv:2607.09385v1 Announce Type: cross Abstract: The growing adoption of large language model-based agents within operating system workflows has increased the importance of energy-efficient inference on laptop-class systems-on-chip (SoCs). While cloud offloading remains common, …
arXiv:2607.09052v1 Announce Type: new Abstract: Block sparse attention is a hardware friendly way to alleviate the key-value (KV) cache read bottleneck in large language models (LLMs). However, it is not prevalent among leading open-weight LLMs, which rely instead on dense attent…
The growing adoption of large language model-based agents within operating system workflows has increased the importance of energy-efficient inference on laptop-class systems-on-chip (SoCs). While cloud offloading remains common, it introduces reliability and privacy concerns tha…
arXiv:2607.07724v1 Announce Type: cross Abstract: Block-sparse attention scales long-context language models by replacing the O(N^2) softmax with a per-query top-k selection over key blocks. This cutoff is myopic: when the k-th and (k+1)-th blocks are nearly tied in score, the se…
arXiv:2601.12145v3 Announce Type: replace Abstract: Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass disperses as sequence lengths increase. We tackle t…
Block sparse attention is a hardware friendly way to alleviate the key-value (KV) cache read bottleneck in large language models (LLMs). However, it is not prevalent among leading open-weight LLMs, which rely instead on dense attention or fine-grained selection, thereby motivatin…
arXiv cs.CL
TIER_1English(EN)·Jo\~ao Monteiro, Michal Klein, Pierre Ablin, Marco Cuturi·
arXiv:2605.09778v2 Announce Type: replace-cross Abstract: Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token. For a given context (a book, a manual, a legal corpus) the attention output is a deterministic f…
arXiv:2602.05305v3 Announce Type: replace-cross Abstract: Generating long-form content, such as minute-long videos and extended texts, is increasingly important for modern generative models. Block diffusion improves inference efficiency via KV caching and block-wise causal infere…
arXiv cs.AI
TIER_1English(EN)·Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang·
arXiv:2607.02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all exist…
Hierarchical Landmark Sparse Attention enables efficient long-context language modeling by learning chunk selection end-to-end, achieving performance comparable to full attention while extrapolating beyond training context lengths.
arXiv:2607.00434v1 Announce Type: cross Abstract: Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning. We claim th…
arXiv:2508.18224v3 Announce Type: replace-cross Abstract: Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language models (LLMs). Native Sparse Attention (NSA), o…
Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning. We claim these failures result from a lack of explicit contro…
arXiv:2606.31054v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image. In this paper, we identify an internal signature of hallucination: progressive degradation…
arXiv cs.LG
TIER_1English(EN)·Rupert Mitchell, Kristian Kersting·
arXiv:2509.10406v4 Announce Type: replace Abstract: Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs. We present Multipole Semantic Attention (MuSe), which accelerates 64k-context …
arXiv:2606.30389v1 Announce Type: new Abstract: Dynamic sparse attention (DSA) accelerates long-context LLM decoding by attending to only the top-K KV blocks relevant to each query, but it introduces a serialized selection-to-attention dependency that emerges as a new latency bot…
arXiv:2606.30562v1 Announce Type: new Abstract: Hybrid attention models improve long-context efficiency by retaining only a subset of full-attention layers and replacing the remaining layers with linear attention. However, the effectiveness of Transformer-to-hybrid conversion cri…
arXiv:2606.28876v1 Announce Type: new Abstract: Long-context language models often conflate two different goals: compressing history into an efficient state, and maintaining reliable long-term memory. Linear, recurrent, and sparse attention reduce the cost of processing long sequ…
arXiv:2606.28560v1 Announce Type: new Abstract: We study sparse self-attention in which each query attends to a dense local window plus a set of Fibonacci-spaced offsets, with a per-layer scalar alpha that compresses or expands the spacing. Across 21 language models trained under…
arXiv:2606.29844v1 Announce Type: cross Abstract: The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios. To improve efficien…
Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image. In this paper, we identify an internal signature of hallucination: progressive degradation of text-to-image cross-attention during generatio…
Hybrid attention models improve long-context efficiency by retaining only a subset of full-attention layers and replacing the remaining layers with linear attention. However, the effectiveness of Transformer-to-hybrid conversion critically depends on which layers preserve full at…
Dynamic sparse attention (DSA) accelerates long-context LLM decoding by attending to only the top-K KV blocks relevant to each query, but it introduces a serialized selection-to-attention dependency that emerges as a new latency bottleneck. We present PRR, a speculate-reuse-repai…
The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios. To improve efficiency, existing approaches often enforce rigid struct…
arXiv cs.LG
TIER_1English(EN)·Yuzhen Mao, Michael Y. Li, Emily B. Fox·
arXiv:2604.20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce Simplified Sparse Attention (SSA), a simpler approach to sparse attention that requires no ar…
arXiv:2510.01718v2 Announce Type: replace Abstract: Attention is a core operation in large language models (LLMs). We present BD Attention (BDA), a lossless algorithmic reformulation of attention. BDA is enabled by a simple matrix identity from Basis Decomposition (BD), which res…
FlashMorph is an efficient layer selection method that formulates hybrid layer selection as a budget-constrained optimization problem, using morphable models and linearization regularization to improve long-context efficiency in Transformers.
Simplified Sparse Attention (SSA) reduces long-context inference costs through gist token-based attention masking during pretraining, enabling efficient chunk selection at inference time without architectural modifications.
Recurrent delta-rule models keep a fixed-size state matrix S (d_v x d_k) that compresses all past context. The state of the art (GDN-2) gates this update with element-wise matrix erase/write masks. This is powerful but has two defects. First, both gates are computed from the inco…
arXiv cs.CV
TIER_1English(EN)·Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubi\'c, Davide Scaramuzza, Wenhui Wang·
arXiv:2603.06228v2 Announce Type: replace Abstract: Event cameras provide sequential visual data with spatial sparsity and high temporal resolution, making them attractive for low-latency object detection. Existing asynchronous event-based neural networks exploit this low-latency…
arXiv:2505.16157v2 Announce Type: replace Abstract: Transformer-based models have made remarkable progress in image restoration (IR) tasks. However, the quadratic complexity of self-attention in Transformer hinders its applicability to high-resolution images. Existing methods mit…
X — Fireworks (inference infra)
TIER_1English(EN)·FireworksAI_HQ·
Long-context sparse attention has a catch: data-dependent block selection wrecks memory access kills speed.
Our @MiniMax_AI M3 kernel on Blackwell answers it. KV-stationary, each block read once, ~980 TFLOP/s on a B200.
See the breakdown here →
https://t.co/LC0zTIKRia https://t…
arXiv cs.CV
TIER_1English(EN)·Fatimah Zohra, Chen Zhao, Shuming Liu, Bernard Ghanem·
arXiv:2607.07135v1 Announce Type: new Abstract: Contrastive Language-Image Pre-training (CLIP) relies on softmax-based self-attention, a strictly positive distribution that assigns probability mass to every pair of tokens-even semantically irrelevant ones. While these dense softm…
Contrastive Language-Image Pre-training (CLIP) relies on softmax-based self-attention, a strictly positive distribution that assigns probability mass to every pair of tokens-even semantically irrelevant ones. While these dense softmax weights are effective for gathering broad con…
arXiv cs.CV
TIER_1English(EN)·Dmitrii Mikhailov, Aleksey Letunovskiy, Maria Kovaleva, Vladimir Arkhipkin, Vladimir Korviakov, Vladimir Polovnikov, Viacheslav Vasilev, Evelina Sidorova, Denis Dimitrov·
arXiv:2507.13546v2 Announce Type: replace Abstract: Recent progress in transformer-based architectures has demonstrated remarkable success in video generation tasks. However, the quadratic complexity of full attention mechanisms remains a critical bottleneck, particularly for hig…
arXiv:2607.01707v1 Announce Type: new Abstract: Large vision-language models (LVLMs) exhibit strong reasoning ability but suffer from visual forgetting during long-horizon decoding, where attention progressively drifts away from visual evidence. Existing methods largely treat thi…
arXiv:2511.14137v3 Announce Type: replace Abstract: Convolutional Neural Networks and Vision Transformers are the two dominant architectural families in computer vision, defined by spatially local convolution and global self-attention respectively. Despite their apparent differen…
Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Atten…
arXiv cs.CV
TIER_1English(EN)·Sanghyun Jo, Wooyeol Lee, Ziseok Lee, Jonghyun Choi, Jaesik Park, Kyungsu Kim·
arXiv:2505.20935v3 Announce Type: replace Abstract: Recent open-weight text-to-image (T2I) diffusion models still struggle with multi-instance prompts, often omitting or merging instances and mixing semantics among similar objects. We trace these failures to early denoising steps…
Fireworks built a KV-stationary sparse-attention kernel for MiniMax M3 on NVIDIA Blackwell (SM100), reaching ~980 TFLOP/s: 1.9–2.4× a query-stationary baseline and ~1.6× open-source MSA. The post walks through the Q-outer vs KV-outer design space, an I/O roofline with the reuse c…
<p>Meituan has released LongCat-2.0, a 1.6 trillion-parameter Mixture-of-Experts model that activates about 48 billion parameters per token. It pairs a native 1-million-token context, built on LongCat Sparse Attention, with training and serving run end-to-end on domestic AI ASIC …
Medium — Claude tag
TIER_1English(EN)·Matthew DaCosta·
<!-- SC_OFF --><div class="md"><p>I’ve been working on a custom Rust + CUDA attention-transformer engine (GAE + ATE + WNSM + reversible training) aimed at determinism and real energy efficiency.</p> <p>Latest sustained numbers on H100 NVL (28-layer 7B-class stack, continuous batc…
dev.to — LLM tag
TIER_1English(EN)·Devanshu Biswas·
<p>If you have ever tried to push a transformer to a longer context and watched your GPU run out of memory, you have met the real bottleneck in attention. It is not the number of multiplications. It is the giant matrix that attention wants to write to memory. Flash Attention is t…
<p>MiniMax-M3 is live on <strong>DEVUP AI</strong> — routable through the same OpenAI-compatible gateway as our other 170+ models, billed in DZD via Edahabia/CIB. Model ID: <code>MiniMaxAI/MiniMax-M3</code>.</p> <p>Most launch-day coverage repeated MiniMax's press numbers without…
