AI models struggle with marathon tasks, revealing benchmark limitations · 1 source tracked

New benchmarks reveal a significant gap between AI models' performance on short, single-session tasks and their ability to handle long, multi-hour operations. While models like GLM-5.2 and GPT-5.5 excel on benchmarks like SWE-bench, their success rate drops dramatically on tasks like SWE-Marathon, which require sustained performance over many hours and millions of tokens. This disparity highlights that current benchmarks may not accurately reflect real-world agent capabilities, and that the true challenge lies in building robust systems with effective self-verification and recovery mechanisms rather than solely focusing on model weights. AI

IMPACT Highlights the need for more realistic benchmarks and robust system design for AI agents to handle complex, long-duration tasks.

RANK_REASON The item discusses new benchmarks and their implications for AI model performance, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• GLM-5.2
• GPT-5.5
• Opus 4.7
• Opus 4.8
• SWE-bench Pro
• SWE-Marathon

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →