PulseAugur
EN
LIVE 20:12:21

New Chess Benchmark Exposes Transformer State-Tracking Failures

By PulseAugur Editorial · [2 sources] ·

Researchers have introduced Chess-World-Model, a new benchmark designed to evaluate the state-tracking capabilities of world models. This benchmark utilizes a dataset of 10 million chess games to test a model's ability to predict the exact board state after a sequence of moves. The study found that recurrent neural network architectures, such as SLiCE, Mamba-3, and Gated DeltaNet, significantly outperformed traditional Transformers on this task, particularly when dealing with out-of-distribution data generated from random play. The research highlights that model scale alone does not guarantee improved state-tracking performance, as failures can remain hidden without specific out-of-distribution testing. AI

IMPACT This benchmark may drive development of more robust world models capable of precise state tracking, potentially impacting AI agents and simulations.

RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating AI models.

Read on arXiv cs.LG →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

New Chess Benchmark Exposes Transformer State-Tracking Failures

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Benjamin Walker, Terry Lyons ·

    Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences

    arXiv:2605.30100v1 Announce Type: new Abstract: World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updat…

  2. arXiv cs.LG TIER_1 English(EN) · Terry Lyons ·

    Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences

    World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updates in realistic domains. We introduce Chess-Worl…

RELATED ENTITIES

RELATED TOPICS