PulseAugur
LIVE 08:29:28
research · [9 sources] ·
0
research

Researchers explore efficient transformers via attention control and algorithmic capture

By PulseAugur Editorial · Summary by gemini-2.5-flash-lite from 9 sources

Researchers are exploring methods to enhance transformer efficiency and understanding. One paper introduces Budgeted Attention Allocation, a head-gating mechanism that allows for cost-quality trade-offs. Another study defines algorithmic capture in transformers and analyzes their computational complexity, suggesting an inductive bias against higher-complexity procedures. Additionally, work on local attention in transformers demonstrates its expressive power and complementarity with global attention, potentially improving model quality. Finally, research investigates how attention sinks can lead to gradient sinks during backpropagation, with massive activations acting as regulators. AI

Summary written by gemini-2.5-flash-lite from 9 sources. How we write summaries →

IMPACT These studies offer theoretical and empirical insights into transformer efficiency, computational complexity, and training dynamics, potentially guiding future model development.

RANK_REASON Multiple arXiv papers present novel research on transformer architectures, efficiency, and computational properties.

Read on arXiv cs.CL →

COVERAGE [9]

  1. arXiv cs.LG TIER_1 · Amrit Nidhi ·

    Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers

    arXiv:2605.05697v1 Announce Type: new Abstract: Transformers usually expose one inference cost per trained model, while deployed systems often need multiple cost-quality operating points. We study Budgeted Attention Allocation, a monotone head-gating mechanism conditioned on a re…

  2. arXiv cs.LG TIER_1 · Orit Davidovich, Zohar Ringel ·

    Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers

    arXiv:2603.11161v2 Announce Type: replace Abstract: We formally define algorithmic capture of combinatorial tasks as the ability of a transformer to extrapolate to arbitrary task sizes with controllable error and logarithmic sample adaptation, providing a sharp scaling criterion …

  3. arXiv cs.LG TIER_1 · Lena Ehrmuth, Laura Strieker ·

    Average Attention Transformers and Arithmetic Circuits

    arXiv:2605.04683v1 Announce Type: cross Abstract: We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circuits if they are given as an input to an encoder. …

  4. arXiv cs.LG TIER_1 · Yihong Chen, Zhouchen Lin, Quanming Yao ·

    Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

    arXiv:2603.17771v2 Announce Type: replace Abstract: Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing explanations have largely focused on the forward pass, yet in pre-norm Transformers, large residual-stream norms…

  5. arXiv cs.AI TIER_1 · Laura Strieker ·

    Average Attention Transformers and Arithmetic Circuits

    We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circuits if they are given as an input to an encoder. The circuit families that can be simulated this wa…

  6. arXiv cs.LG TIER_1 · Stephen J. Thomas ·

    Cascade Token Selection for Transformer Attention Acceleration

    arXiv:2605.03110v1 Announce Type: new Abstract: A method is presented for reducing the cost of representative token selection in transformer attention layers by exploiting the coherence of the representative set across depth. Activation Decorrelation Attention (ADA) selects $r \l…

  7. arXiv cs.CL TIER_1 · Jiaoda Li, Ryan Cotterell ·

    Characterizing the Expressivity of Local Attention in Transformers

    arXiv:2605.00768v1 Announce Type: new Abstract: The transformer is the most popular neural architecture for language modeling. The cornerstone of the transformer is its global attention mechanism, which lets the model aggregate information from all preceding tokens before generat…

  8. arXiv cs.CL TIER_1 · Ryan Cotterell ·

    Characterizing the Expressivity of Local Attention in Transformers

    The transformer is the most popular neural architecture for language modeling. The cornerstone of the transformer is its global attention mechanism, which lets the model aggregate information from all preceding tokens before generating the next token. One common variant of attent…

  9. dev.to — LLM tag TIER_1 · Rijul Rajesh ·

    Understanding Decoder-Only Transformers Part 1: Masked Self-Attention

    <h2> Decoder-Only Transformers </h2> <p>In this article, we will explore <strong>decoder-only transformers</strong>.</p> <p>Decoder-only transformers are a specific type of transformer architecture used in systems like ChatGPT.</p> <h2> Masked Self-Attention </h2> <p>Decoder-only…

RELATED ENTITIES

RELATED TOPICS