A new metric called pass^k is proposed to better evaluate the reliability of AI agents, particularly in user-facing applications. Unlike pass@1, which measures the success rate of a single attempt, pass^k assesses how consistently an agent performs a task across multiple identical trials. This is crucial for applications like customer service where repeated interactions demand high reliability. For instance, GPT-4o achieved a pass^1 of 61% on a retail task but dropped to 25% for pass^8, highlighting a significant gap between single-run success and true consistency. AI
IMPACT Highlights the need for consistency metrics beyond single-run success rates for AI agents, impacting how their performance is evaluated and deployed.
RANK_REASON Introduces a new metric for evaluating AI agent reliability. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →