A new Harness Leaderboard v0.1.9, compiled by Joel Niklaus and hosted on Hugging Face Space, evaluates 11 different coding-agent harnesses using the same models and tasks. The results show a significant variance in performance, with macro-average resolve rates ranging from 42.7% to 52.5%. The leaderboard highlights that while models are a primary driver of performance, the choice of harness also plays a crucial role, influencing task outcomes by up to 27.1%. It also identifies practical issues like incorrect context windows and ineffective history compaction as bugs found in various harnesses. AI
IMPACT Highlights the significant impact of harness selection on LLM performance for coding tasks, suggesting engineers should carefully consider harness choice alongside model selection.
RANK_REASON The item describes a benchmark and leaderboard for evaluating different coding-agent harnesses, which falls under research and evaluation of AI systems.
- Claude Code
- Codex
- DeepSeek Harness
- Flintshire
- Grok Build
- Harness Leaderboard v0.1.9
- Hermes Agent
- Hugging Face Space
- Joel Niklaus
- Kimi Code
- mini-SWE-agent
- OpenClaw
- OpenCode
- Qwen Code
- Pi
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →