PulseAugur
EN
LIVE 01:20:46

AI agent performance hinges on harness, not just model, study finds

A recent experiment involving 27,280 agent runs and 83 billion tokens revealed that the surrounding harness configuration, rather than the underlying language model, significantly impacts performance and cost. Across 22 different harness setups and 5 models, the median harness moved 11 places in cost-per-task rankings when the model changed, indicating that prompt engineering, tool usage, and stopping conditions are more critical than the model itself. The study found that expensive harnesses often passed less often than cheaper ones, and a large portion of costs were attributed to failed trials and timeouts, highlighting the need for efficient loop design and review processes. AI

IMPACT Highlights that optimizing AI agent performance and cost requires focusing on prompt engineering and loop design, not just model selection.

RANK_REASON The item details a novel experiment and analysis of AI agent performance, presenting findings and a methodology. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

AI agent performance hinges on harness, not just model, study finds

How we ranked this

Signal score
56 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The item details a novel experiment and analysis of AI agent performance, presenting findings and a methodology. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
product, infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · Vinny Barreca ·

    27,280 Agent Runs Prove Your Harness Matters More Than Your Model

    <p>27,280 agent runs. 83 billion tokens. 22 harness configurations, 5 models, 62 terminal tasks, 4 attempts each.</p> <p>The headline is not which model won. It is that the same harness moved eleven of twenty-two places when the model underneath it changed.</p> <p>Codex CLI ranke…