Qwen2.5-7B-Instruct accuracy drops with JSON constraints, but can be recovered

A study on the Qwen2.5-7B-Instruct model revealed that enforcing strict JSON output schemas, while ensuring compliance, can reduce mathematical accuracy by up to 18.4 percentage points. This reduction was attributed to the model struggling with specific data type requirements, such as outputting a numeric answer as a string. However, by compiling the contract before generation and allowing the model to use native JSON integers internally before converting to the required string format, accuracy was recovered by 14.3 percentage points while maintaining 100% schema validity. AI

IMPACT Highlights the trade-offs between strict output compliance and inherent model reasoning capabilities, informing best practices for structured data generation.

RANK_REASON The item details a controlled study on a specific model's performance with output constraints, presenting findings and methodology. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• GSM8K
• JSON
• Qwen2.5-7B-Instruct
• XGrammar

• model release
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →