JSON schema field order critically impacts LLM output, developer finds

A developer discovered that the order of keys in a JSON schema significantly impacts the output of large language models when using constrained decoding. When a cache helper function sorted the JSON schema keys alphabetically, the model began assigning categories before generating reasoning, leading to a notable decrease in accuracy. This issue arises because autoregressive models generate text token by token, and later tokens cannot influence earlier decisions. The developer found that placing the 'reasoning' field before the 'category' field in the schema improved the model's accuracy from 81.8% to 90.2%. AI

IMPACT Highlights a subtle but critical factor in prompt engineering and constrained decoding for LLMs, impacting reliability and accuracy.

RANK_REASON Developer's personal blog post detailing a specific technical finding about LLM behavior.

Read on dev.to — LLM tag →

• json-schema
• llama.cpp
• pydantic
• Triage
• vLLM

• product
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →