Researchers explore efficient transformers via attention control and algorithmic capture

COVERAGE [9]

1. arXiv cs.LG TIER_1 English(EN) · Amrit Nidhi · 2026-05-08 04:00

   Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers

   arXiv:2605.05697v1 Announce Type: new Abstract: Transformers usually expose one inference cost per trained model, while deployed systems often need multiple cost-quality operating points. We study Budgeted Attention Allocation, a monotone head-gating mechanism conditioned on a re…

2. arXiv cs.LG TIER_1 English(EN) · Orit Davidovich, Zohar Ringel · 2026-05-08 04:00

   Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers

   arXiv:2603.11161v2 Announce Type: replace Abstract: We formally define algorithmic capture of combinatorial tasks as the ability of a transformer to extrapolate to arbitrary task sizes with controllable error and logarithmic sample adaptation, providing a sharp scaling criterion …

3. arXiv cs.LG TIER_1 English(EN) · Lena Ehrmuth, Laura Strieker · 2026-05-07 04:00

   Average Attention Transformers and Arithmetic Circuits

   arXiv:2605.04683v1 Announce Type: cross Abstract: We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circuits if they are given as an input to an encoder. …

4. arXiv cs.LG TIER_1 English(EN) · Yihong Chen, Zhouchen Lin, Quanming Yao · 2026-05-07 04:00

   Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

   arXiv:2603.17771v2 Announce Type: replace Abstract: Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing explanations have largely focused on the forward pass, yet in pre-norm Transformers, large residual-stream norms…

5. arXiv cs.AI TIER_1 English(EN) · Laura Strieker · 2026-05-06 09:35

   Average Attention Transformers and Arithmetic Circuits

   We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circuits if they are given as an input to an encoder. The circuit families that can be simulated this wa…

6. arXiv cs.LG TIER_1 English(EN) · Stephen J. Thomas · 2026-05-06 04:00

   Cascade Token Selection for Transformer Attention Acceleration

   arXiv:2605.03110v1 Announce Type: new Abstract: A method is presented for reducing the cost of representative token selection in transformer attention layers by exploiting the coherence of the representative set across depth. Activation Decorrelation Attention (ADA) selects $r \l…

7. arXiv cs.CL TIER_1 English(EN) · Jiaoda Li, Ryan Cotterell · 2026-05-04 04:00

   Characterizing the Expressivity of Local Attention in Transformers

   arXiv:2605.00768v1 Announce Type: new Abstract: The transformer is the most popular neural architecture for language modeling. The cornerstone of the transformer is its global attention mechanism, which lets the model aggregate information from all preceding tokens before generat…

8. arXiv cs.CL TIER_1 English(EN) · Ryan Cotterell · 2026-05-01 16:30

   Characterizing the Expressivity of Local Attention in Transformers

   The transformer is the most popular neural architecture for language modeling. The cornerstone of the transformer is its global attention mechanism, which lets the model aggregate information from all preceding tokens before generating the next token. One common variant of attent…

9. dev.to — LLM tag TIER_1 English(EN) · Rijul Rajesh · 2026-05-05 19:25

   Understanding Decoder-Only Transformers Part 1: Masked Self-Attention

   <h2> Decoder-Only Transformers </h2> <p>In this article, we will explore <strong>decoder-only transformers</strong>.</p> <p>Decoder-only transformers are a specific type of transformer architecture used in systems like ChatGPT.</p> <h2> Masked Self-Attention </h2> <p>Decoder-only…