A recent experiment involving 27,280 agent runs and 83 billion tokens revealed that the surrounding harness configuration, rather than the underlying language model, significantly impacts performance and cost. Across 22 different harness setups and 5 models, the median harness moved 11 places in cost-per-task rankings when the model changed, indicating that prompt engineering, tool usage, and stopping conditions are more critical than the model itself. The study found that expensive harnesses often passed less often than cheaper ones, and a large portion of costs were attributed to failed trials and timeouts, highlighting the need for efficient loop design and review processes. AI
IMPACT Highlights that optimizing AI agent performance and cost requires focusing on prompt engineering and loop design, not just model selection.
RANK_REASON The item details a novel experiment and analysis of AI agent performance, presenting findings and a methodology. [lever_c_demoted from research: ic=1 ai=1.0]
- Alma
- Codex CLI
- DeepSeek
- DeepSeek Harness: Creator
- FrontierHarness Eval v1.1
- GLM 5.3
- glm-5.3(-flash)
- GLM Flash
- Kimi k3
- mini-SWE-agent
- Muse
- oh-my-pi
- OpenCode v1
- Raspberry Pi
- ZCode
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →