PulseAugur
EN
LIVE 06:48:04

New benchmark reveals LLMs struggle with precise text structure reconstruction

By PulseAugur Editorial · [1 sources] ·

Researchers have developed OrderProbe, a new benchmark designed to evaluate how well large language models (LLMs) can reconstruct the precise structural order of text. Unlike previous methods that allowed for multiple correct reorderings, OrderProbe uses fixed four-character expressions in Chinese, Japanese, and Korean to enable exact-match scoring. Experiments on twelve LLMs revealed that even advanced models struggle with this task, often achieving less than 35% accuracy in zero-shot recovery, indicating a gap between semantic understanding and precise structural reconstruction. AI

IMPACT Highlights a key limitation in current LLMs, suggesting a need for improved architectural designs focused on structural integrity.

RANK_REASON The cluster contains an academic paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

New benchmark reveals LLMs struggle with precise text structure reconstruction

How we ranked this

Signal score
27 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The cluster contains an academic paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. arXiv cs.CL TIER_1 English(EN) · Zhaolu Kang, Yingjie He, Kehan Jiang, Leqi Zheng, Jiachen Qian, Qianyuan Zhang, Chunlei Meng, Yujie Feng, Yuan Wang, Stephen Dou, Aming Wu, Pengxiang Zhao, Jiaxin Liu, Guansu Wang, Zeyu Zhang, Lei Wang, Qishi Zhan, Xiaomin He, Meisheng Zhang, Jianyuan Ni… ·

    How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

    arXiv:2601.08626v4 Announce Type: replace Abstract: Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is difficult to evaluate automatically bec…

RELATED ENTITIES

RELATED TOPICS