A recent benchmark evaluated six large language models on their ability to extract structured data, specifically JSON, from customer support emails. The analysis found that Anthropic's Claude Haiku 4.5 offered the best value, achieving high accuracy at a significantly lower cost compared to more powerful models. While Gemini 2.5 Flash was fast and inexpensive, it struggled with accuracy, particularly in hallucinating data. The study suggests using Haiku for most extraction tasks, Sonnet for more complex reasoning, and avoiding more expensive frontier models for simple data extraction. AI
IMPACT Identifies the most cost-effective LLM for structured data extraction, guiding developers on model selection for production features.
RANK_REASON The cluster describes a benchmark comparing LLM performance on a specific task, rather than a new model release or major industry event. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
