Researchers have developed a new protocol for evaluating agentic Video Question Answering (VideoQA) systems, focusing on both accuracy and cost. This method pairs two systems to jointly assess differences in correctness and inference effort, categorizing outcomes into six groups based on these metrics. When applied to the Dynamic-SAGE framework against the SAGE baseline on SAGE-Bench, the protocol revealed that Dynamic-SAGE improved accuracy by 7.5 points while reducing reasoning turns and tool calls by approximately 28%. However, it also increased token usage by 34% and cost by 26%, indicating a shift rather than a reduction in inference cost. AI
IMPACT This new auditing protocol could lead to more efficient AI agents by providing a clearer picture of cost-performance trade-offs.
RANK_REASON The cluster contains an academic paper detailing a new methodology for evaluating AI systems. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
