Toolkit Labs has released findings from a benchmark of JSON parsers designed to handle malformed output from large language models. The study focused on 25 cases of truncated JSON, where a model's output is cut off mid-stream. Out of 21 parsers tested across Python and Node.js runtimes, 12 scored zero, failing to recover any of the truncated outputs. The company's own parser, jsonshim, performed comparably to json-repair, recovering 23 out of 25 truncated cases. AI
IMPACT Highlights challenges in reliably parsing LLM output, potentially impacting downstream applications that rely on structured data.
RANK_REASON The item details a benchmark of software parsers for handling malformed LLM output, including a public dataset. [lever_c_demoted from research: ic=1 ai=0.7]
- commentjson
- hjson
- json5
- json-loose
- json-repair
- jsonshim
- MALFORMED-300
- node v25.8.2
- pyjson5
- Python 3.12.3
- simplejson
- Toolkit Labs
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
