LLMs struggle to predict AI agent job success without specific limitations

Researchers have developed PAYWORTHY, a new dataset comprising 1,494 freelance job postings, to evaluate the ability of large language models (LLMs) to predict the success rate of AI agents on these tasks. When informed about an agent's limitations, LLMs demonstrated strong predictive capabilities, achieving AUC scores between 0.66 and 0.86. However, when presented with a generic agent description, the same LLMs performed at a random chance level (AUC 0.45-0.56). This predictive accuracy was consistent across various model families, including Claude, GLM, Kimi, and Qwen, and the LLM forecasts also aligned with human judgments. AI

IMPACT Highlights the critical need for specific agent capabilities to be defined for LLMs to accurately assess task success, impacting agent deployment and evaluation frameworks.

RANK_REASON The cluster contains a research paper detailing a new dataset and evaluation methodology for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Claude
• General Language Model
• AI agent
• LLMs
• Mohammad R. Rezaei
• PAYWORTHY
• Qwen

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →