PulseAugur
EN
LIVE 18:30:30

Frontier LLMs struggle with state tracking on new StateBench evaluation

By PulseAugur Editorial · [1 sources] ·

A new benchmark called StateBench has revealed that leading large language models struggle with tracking state changes in long texts. Frontier models like GPT-6 Sol and Claude Opus 5.5 achieved accuracy rates of 13% or lower on the most challenging, extended scenarios. This evaluation highlights a significant limitation in current LLM capabilities regarding long-term context and state tracking. AI

IMPACT Highlights a critical limitation in LLMs' ability to maintain context and track state over extended interactions, potentially impacting applications requiring long-term memory.

RANK_REASON The item describes a new benchmark for evaluating LLMs, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Bluesky Jetstream — AI desk →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

Frontier LLMs struggle with state tracking on new StateBench evaluation

How we ranked this

Signal score
13 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The item describes a new benchmark for evaluating LLMs, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. Bluesky Jetstream — AI desk TIER_1 English(EN) · ai2.bsky.social ·

    Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment?

    Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment? On our new StateBench eval, frontier models including GPT-6 Sol & Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems. 🧵 buff.ly/NF3iDYc

    LINKS github.com/…/state-bench

RELATED ENTITIES

RELATED TOPICS