A new dataset has been created to track the cost of AI agent performance on various benchmarks, addressing a gap in existing leaderboards that primarily focus on scores. This dataset connects agent configurations, benchmark tasks, verified success, and the recorded cost of each run. It reveals a significant price disparity, with costs ranging from $0.03 to over $1,600 for systems that appear comparable on agent leaderboards. The analysis highlights that for tasks with cheap verification and retry capabilities, lower-cost configurations can be more competitive than score-based rankings alone suggest. AI
IMPACT Highlights the significant cost variations in AI agent performance, suggesting cost-effectiveness should be a key metric alongside accuracy.
RANK_REASON New dataset and analysis of AI agent performance costs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
