Constrained decoding cuts LLM math accuracy by 18% while fixing JSON output

A controlled experiment with the Qwen2.5-7B model revealed that using constrained decoding to enforce JSON schema compliance significantly reduced mathematical accuracy. While tools like Outlines and XGrammar successfully improved JSON compliance from 0% to 100%, they also caused a 18.4 percentage-point drop in mathematical accuracy, from 79.6% to 61.2%. This suggests that the process of enforcing output structure can interfere with the model's reasoning capabilities, indicating that syntax and semantic correctness are not always independent. AI

IMPACT Enforcing strict output formats like JSON may negatively impact LLM reasoning and accuracy on complex tasks.

RANK_REASON Controlled experiment on LLM output formatting and accuracy. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• json-schema
• Qwen2.5-7B
• XGrammar

• model release
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →