A new benchmark, FrontierHarness Eval v1.1, reveals that while the AI model itself accounts for 92.8% of cost variation in coding agents, the harness configuration can increase costs by up to 5x for the same model. The study found that a significant portion of spending, 64.5%, goes towards trials that ultimately fail or time out, with agents often providing confident but incorrect final answers. This suggests that the cost of AI coding agents is more influenced by the number of steps taken and the verification process rather than solely the model's capabilities. AI
IMPACT Highlights that AI agent costs are heavily influenced by harness configuration and verification, not just the model itself, impacting operational efficiency.
RANK_REASON New benchmark release with detailed analysis of AI agent performance and cost factors. [lever_c_demoted from research: ic=1 ai=1.0]
- Cline CLI
- DeepSeek Harness
- DeepSeek V4.1 Flash
- FrontierHarness Eval v1.1
- GLM 5.3
- Kimi k3
- Muse
- openCode
- Runta
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →