Debugger vs. Bash: AI Coding Agents Show Bash Outperforms Debuggers on SWE-bench

A new study compares the effectiveness of interactive debuggers versus traditional edit-execute-observe loops for AI coding agents. Researchers evaluated two models, GPT-OSS-120B and Nemotron-3-Ultra-550B, on the SWE-bench benchmark using a novel metric called pass$_X$@$k$. The findings suggest that while interactive debuggers offer live access to runtime state, two separate runs using bash commands outperformed a combination of one bash and one debugger rollout. The study emphasizes the need to account for stochastic variance when assessing agent and tool diversity in AI development. AI

IMPACT Highlights the importance of accounting for stochastic variance in evaluating AI agent tools and suggests bash-based debugging may be more effective than interactive debuggers for current AI coding agents.

RANK_REASON Research paper detailing a new metric and experimental results on AI coding agents. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• GPT-OSS 120B
• Nemotron 3 Ultra-550B
• pass$_X$@$2$
• pass$_X$@$k$
• SWE-bench

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →