A recent benchmark comparing 10 large language models revealed that the cheapest model per token is not necessarily the most cost-effective for completing tasks. The study, which involved 2,400 runs across 40 command-line tasks, introduced a "cost per successful task" metric that accounts for retries and failed attempts. This metric showed that while models like GPT-OSS 120B were significantly cheaper per successful task, they had lower pass rates and coverage compared to premium models like GPT-5.5 and Kimi K3. The findings suggest that the optimal model choice depends heavily on the specific task and desired reliability, rather than just the per-token pricing. AI
IMPACT Highlights the importance of evaluating LLMs based on task completion cost and reliability, rather than just token pricing, for practical AI applications.
RANK_REASON Analysis of LLM performance and cost-effectiveness using a novel metric. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
