LLM agents pass JSON schema checks but fail 1 in 5 orders on new benchmark

By PulseAugur Editorial · [1 sources] · 2026-07-25 12:48

A new benchmark called New OrderBench has revealed that while LLM agents can achieve 100% success in JSON schema checks, they still fail to correctly process one in five orders. This indicates that adherence to structural validity does not guarantee semantic accuracy in their operations. AI

IMPACT Highlights the gap between structural compliance and functional correctness in LLM agents, suggesting a need for more robust semantic evaluation.

RANK_REASON New benchmark evaluation of LLM agent capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Mastodon — fosstodon.org →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

LLM agents pass JSON schema checks but fail 1 in 5 orders on new benchmark

COVERAGE [1]

Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] · 2026-07-25 12:48

LLM agents pass 100% JSON schema checks, still fail 1 in 5 orders New OrderBench benchmark runs 2,400 calls across four open models and finds schema validity al

LLM agents pass 100% JSON schema checks, still fail 1 in 5 orders New OrderBench benchmark runs 2,400 calls across four open models and finds schema validity alone doesn't guarantee semantic correctness. https://www. notatechguy.com/llm-agents-pas s-100-json-schema-checks-still-f…

LINKS notatechguy.com/llm-agents-pass-100-json-…

COVERAGE [1]

LLM agents pass 100% JSON schema checks, still fail 1 in 5 orders New OrderBench benchmark runs 2,400 calls across four open models and finds schema validity al

RELATED ENTITIES

RELATED TOPICS