PulseAugur
EN
LIVE 11:33:50

Researchers find Transformers know counts but struggle to output them

By PulseAugur Editorial · [2 sources] ·

A new paper identifies a specific bottleneck in Transformer models that hinders their ability to perform counting tasks. Researchers found that while models like Pythia, Qwen3, and Mistral store count information accurately internally, they struggle to translate this information into the correct output tokens. A targeted intervention on attention weights significantly improved the models' ability to generate correct counts in autoregressive tasks, suggesting a geometric misalignment in the output pathway. AI

IMPACT Identifies a specific readout bottleneck in Transformers for counting tasks, potentially guiding future model architectures.

RANK_REASON The cluster contains an academic paper detailing a novel finding about Transformer model limitations.

Read on arXiv cs.CL →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

Researchers find Transformers know counts but struggle to output them

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
The cluster contains an academic paper detailing a novel finding about Transformer model limitations.
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
132 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Gabriel Garcia ·

    The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It

    arXiv:2605.03258v1 Announce Type: new Abstract: Large language models often fail at simple counting tasks, even when the items to count are explicitly present in the prompt. We investigate whether this failure occurs because transformers do not represent counts internally, or bec…

  2. arXiv cs.CL TIER_1 English(EN) · Gabriel Garcia ·

    The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It

    Large language models often fail at simple counting tasks, even when the items to count are explicitly present in the prompt. We investigate whether this failure occurs because transformers do not represent counts internally, or because they cannot convert those representations i…

RELATED ENTITIES

RELATED TOPICS