English(EN)Training-free sparse attention based on cumulative energy filtering
新研究探索用于大型语言模型的混合和稀疏注意力机制
作者PulseAugur 编辑部·[19 个来源]·
研究人员正在探索优化大型语言模型中注意力机制的新方法，特别是用于处理长上下文。例如，HydraHead架构沿头轴混合了全注意力（FA）和线性注意力（LA），识别关键的FA头并协调分布差异。另外，StreamKL提供了一种快速且内存高效的方法来计算注意力蒸馏的Kullback-Leibler散度，减少了HBM占用空间，并支持在单个GPU上进行长上下文蒸馏。其他研究包括基于域分解的分层注意力和距离自适应表示，后者为远距离标记分配较低的维度，挑战了统一维度的假设。
AI
arXiv:2606.20097v1 Announce Type: new Abstract: The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the …
arXiv:2606.20005v1 Announce Type: cross Abstract: Attention distillation, which trains one attention distribution to match another by minimizing their Kullback-Leibler (KL) divergence, is widely used in knowledge distillation, model compression, continual learning, and sparse-att…
The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the inherent difficulty of integrating Linear Attent…
Attention distillation, which trains one attention distribution to match another by minimizing their Kullback-Leibler (KL) divergence, is widely used in knowledge distillation, model compression, continual learning, and sparse-attention LLM training. However, existing approaches …
arXiv cs.LG
TIER_1English(EN)·Stephan K\"ohler, Oliver Rheinbach·
arXiv:2606.18525v1 Announce Type: new Abstract: We propose a hierarchical attention mechanism based on two-level overlapping Schwarz domain decomposition. The method is motivated by the observation that two-level Schwarz domain decomposition methods combine local subdomain correc…
arXiv cs.AI
TIER_1English(EN)·Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan·
arXiv:2606.18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its distance from the prediction target. In natural languag…
arXiv:2606.18056v1 Announce Type: new Abstract: Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely on hand-crafted rules or simple post-hoc heuristics…
arXiv:2502.08363v3 Announce Type: replace-cross Abstract: We present Top-Theta (Top-$\theta$) Attention, a training-free method for sparsifying transformer attention during inference. Our key insight is that static, per-head thresholds can be calibrated to retain the desired cons…
Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its distance from the prediction target. In natural language, however, the next word is most strongly influen…
Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely on hand-crafted rules or simple post-hoc heuristics for FA/SWA allocation and offer limited analysi…
arXiv cs.CL
TIER_1English(EN)·Luke McDermott, Robert W. Heath Jr., Rahul Parhi·
arXiv:2505.23666v3 Announce Type: replace Abstract: The per-token cost of transformer inference scales with context length, preventing its application to lifelong in-context learning. Linear attention is an efficient alternative that maintains a constant memory footprint, even on…
Hybrid linear attention models can be improved through a novel initialization technique that enhances conversion from pretrained Transformers by leveraging teacher attention statistics and alignment steps.
arXiv:2512.11784v2 Announce Type: replace-cross Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis. We develop a unified, measure-based framework for studying single-la…
arXiv cs.CV
TIER_1English(EN)·Chunlu Li, Yixuan Pan, Bai Du, Zhenyuan Chen, Yanzhao Li, Hui Dong, Hui Wang, Zhiqiang Zou·
arXiv:2606.16317v1 Announce Type: new Abstract: Sparse attention accelerates Diffusion Transformers (DiTs) for video generation by computing only the important tokens while skipping the rest. The token selection strategy is key to balancing sparsity and accuracy. We formulate the…
Sparse attention accelerates Diffusion Transformers (DiTs) for video generation by computing only the important tokens while skipping the rest. The token selection strategy is key to balancing sparsity and accuracy. We formulate the token filtering process as a dual-goal optimiza…
<p>MiniMax released MSA, a sparse attention built on Grouped Query Attention. A lightweight Index Branch selects Top-k key-value blocks per query and GQA group; the Main Branch attends only to those blocks. It matches GQA on downstream benchmarks while reducing per-token attentio…
<table> <tr><td> <a href="https://www.reddit.com/r/MachineLearning/comments/1ubmybr/i_released_a_softmaxfree_attention_model_at_gpt2/"> <img alt="I released a softmax-free attention model at GPT-2 Medium scale (~354M params, 11.5B tokens): structural sparsity + tile-skipping kern…
🤖 MiniMax's MSA Cuts AI Compute Costs with Sparse Attention MiniMax has developed and deployed a sparse attention method called MiniMax Sparse Attention (MSA) that reduces the computational cost of AI models by up to a fixed O(kB k ) complexity, regardless of context length. This…
