Frontier AI models fail new IT benchmark, scoring below 50%

A new benchmark, ITBench-AA, has been released to evaluate the capabilities of frontier AI models on enterprise IT tasks, specifically focusing on Site Reliability Engineering (SRE). In initial tests, even the most advanced models like Claude Opus 4.7 and GPT-5.5 scored below 50% on diagnosing Kubernetes incidents. The benchmark revealed that models struggle with root-cause analysis, and longer investigation trajectories do not necessarily lead to higher accuracy, with some models over-investigating and identifying false positives. AI

IMPACT Highlights significant limitations in current frontier models for complex, real-world enterprise IT operations, suggesting a need for improved reasoning and diagnostic capabilities.

RANK_REASON The cluster describes the release of a new benchmark for evaluating AI models on specific tasks, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Blog →

• Artificial Analysis
• Claude Opus 4.7
• DeepSeek V4 Pro
• Gemini 3.1 Pro Preview
• Gemini 3.5 Flash
• Gemma 4 31B
• GLM-5.1
• GPT-5.5
• IBM Research
• ITBench-AA
• Kubernetes
• Qwen3.7 Max

• paper
• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →