An experiment comparing a 4B parameter model (Qwen3.5 4B) against a 12B parameter model (Gemma4 12B) in a tool-using agent revealed that the smaller model struggled significantly, passing only 70 out of 96 tasks in the initial test. After a reader pointed out a flaw in the testing methodology, the experiment was re-run with tool errors returned as messages to the model. In this second run, the 4B model passed 89% of tasks, while the 12B model achieved 100% success, indicating a gap of 10.8 percentage points. The smaller model's performance degraded further when faced with compound failures, such as making an incorrect initial tool call followed by an empty lookup result. AI
IMPACT Highlights the performance gap between smaller and larger models in tool-use scenarios, suggesting that returning tool errors to the model can partially close this gap.
RANK_REASON The cluster details an experimental comparison of two specific LLMs on a defined task set, including methodology and results. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
