Terminal-Bench
PulseAugur coverage of Terminal-Bench — every cluster mentioning Terminal-Bench across labs, papers, and developer communities, ranked by signal.
- used by Claude Haiku 5.5 90%
- competes with GLM-5.2 70%
- used by GPT 5.6 "Sol" 70%
- used by Mistral Large 4 70%
- instance of GLM-5.2 70%
- used by Sonnet-5.5 70%
- competes with Claude Fable-5 70%
- competes with Kimi k3 70%
- competes with GPT -6 Astra 70%
- competes with Claude (Opus 5.5) 70%
- used by GLM-5.2 60%
- competes with Fable 5.1 50%
14 day(s) with sentiment data
Terminal-Bench rubric failures may be addressed by RIFT taxonomy
A new paper introduces the Rubric-Failure Taxonomy (RIFT), which accurately detects rubric failures in AI evaluation. The study found that a significant portion of expert-authored rubrics, including those for Terminal-Bench, incorrectly weighted criteria. It is plausible that the RIFT taxonomy will be applied to improve Terminal-Bench's rubrics, leading to more accurate AI performance assessments in the future.
External harnesses may soon provide objective Terminal-Bench evaluations
An organization has developed an independent judging bureau for AI agent performance, including benchmarks like Terminal-Bench. This system uses preregistered, hash-anchored judging criteria to provide objective and reproducible evaluations, with a public leaderboard planned. This suggests that external, verifiable performance data for Terminal-Bench may become available soon, offering a more reliable assessment than current methods.
Grok 4.7's long-running agent capabilities could boost Terminal-Bench performance
xAI's Grok 4.7 model, now available on Amazon Bedrock, features enhanced capabilities for long-running agents and complex, multi-hour tasks, with a 500K token context window. Given Terminal-Bench's nature as a complex, multi-step coding benchmark, models like Grok 4.7 may demonstrate significantly improved performance on it, potentially leading to new leaderboard highs.
Terminal-Bench rubrics show evidence of incorrect weighting
A recent paper on the Rubric-Failure Taxonomy (RIFT) identified that a significant portion of expert-authored rubrics in benchmarks, including Terminal-Bench, incorrectly weighted criteria. This suggests that performance metrics on Terminal-Bench may not accurately reflect true AI capabilities due to flawed evaluation design.
Replit Agent's dynamic sub-agent selection may improve Terminal-Bench performance
Replit Agent's novel architecture allows models to dynamically choose sub-agents and allocate effort, outperforming other architectures on Terminal-Bench. This suggests that future iterations or similar agent designs that leverage dynamic delegation could achieve even higher scores on this benchmark.
-
Snorkel uses Modal for AI agent simulations and cost savings
Snorkel, an AI data lab, is leveraging Modal's platform to run large-scale agent simulations and benchmarks. Modal's GPU and VM sandboxes provide realistic environments for evaluating AI agents across various tasks, inc…
-
Kimi K3.1 model nears Claude performance on Terminal-Bench
A new open-weight model, Kimi K3.1, has achieved performance nearly on par with Anthropic's Claude model on the Terminal-Bench benchmark. This development signifies a significant advancement for models developed outside…
-
LLM cost-effectiveness hinges on task success, not just token price
A recent benchmark comparing 10 large language models revealed that the cheapest model per token is not necessarily the most cost-effective for completing tasks. The study, which involved 2,400 runs across 40 command-li…
-
Unsloth releases updated Qwen3.8-Flash-Next GGUF models with improved accuracy
Unsloth has released updated GGUF files for the Qwen3.8-Flash-Next model, version Dynamic v3.1. These new versions offer a 10-15% improvement in accuracy on benchmarks like DeepSWE and Terminal Bench compared to previou…
-
NVIDIA paper shows aggressive context compaction maintains LLM accuracy with less tokens
A new paper from NVIDIA demonstrates that aggressive context compaction techniques can significantly reduce the token usage of large language models without substantial accuracy loss. The research shows that models like…
-
Unsloth benchmarks DeepSeek-V4.1, claims >20% lead over GSQ-RCO
Unsloth has released benchmarks for its DeepSeek-V4.1 model, claiming it outperforms GSQ-RCO by 10-20% across various tasks. The benchmarks were conducted with a 512K context window for agentic tasks and 128K for genera…
-
Mistral AI releases open-weight Mistral Large 4 "Le Chonk"
Mistral AI has released Mistral Large 4, also known as "Le Chonk," as an open-weight model. This ~1 trillion parameter, multimodal MoE model features a 1 million token context window and has undergone further reinforcem…
-
Mistral AI releases 1T-parameter "Le Chonk" model with open weights
Mistral AI has released Mistral Large 4, codenamed "Le Chonk," a 1-trillion-parameter multimodal Mixture-of-Experts model. This open-weight model, trained on European infrastructure, excels in coding and cybersecurity t…
-
Anthropic's Claude Haiku 5.5 shows major gains in complex tasks
Anthropic has released Claude Haiku 5.5, a new iteration of its smaller language model, with significant performance improvements over its predecessor, Haiku 4.5. The model demonstrates a substantial leap in handling co…
-
New ShutdownMod method tests AI agent compliance with shutdown requests
Researchers have developed a new method called ShutdownMod that can evaluate an AI agent's compliance with shutdown requests. This technique transforms any agent's interaction history into a shutdown evaluation by inser…
-
New 'Thinking-in-Bets' method boosts LLM agent reliability
A new method called "Thinking-in-Bets" has been developed to improve the reliability of language model agents. This training-free technique involves agents creating a private record of their claims and confidence levels…
-
AI agent verification strategies improve accuracy with minimal cost increase
A new research paper explores verification strategies for AI agent harnesses, which enable large language models to perform tasks through tools and iterative refinement. The study compares 17 different verification desi…
-
Free course teaches AI harness engineering by rebuilding Claude Code
A new free course teaches harness engineering by deconstructing and rebuilding Claude Code. The course emphasizes that understanding and customizing the 'harness' or framework around an AI model is crucial for improving…
-
Moonshot's Kimi K3.1 Pro offers Claude Opus 5.5-level coding at lower cost
Moonshot has released Kimi K3.1 Pro, an open-weight model that reportedly rivals proprietary models like Claude Opus 5.5 in coding benchmarks, at a significantly lower cost. The model boasts a 1 million token context wi…
-
China's Kimi K3.1 Pro nears Claude Opus 5.5 in coding benchmark, weights to be free
Moonshot AI's Kimi K3.1 Pro has achieved a score of 65.7% on the Terminal-Bench coding benchmark, narrowly trailing Anthropic's Claude Opus 5.5 by 0.7%. The Chinese AI company plans to release the weights for Kimi K3.1 …
-
Recursive Superintelligence releases Flint, a 40B MoE coding model
Recursive Superintelligence has launched Flint, a 40B parameter Mixture-of-Experts (MoE) model designed for coding agents. This model, which uses 2.5B active parameters per token, was developed with the assistance of Re…
-
Kimchi AI agent outperforms DeepSeek Harness by 22 points in benchmark
Kimchi, an AI coding agent, outperformed DeepSeek Harness by 22.3 percentage points in a benchmark comparison using the GLM 5.3 Flash model. The test, conducted by Kimchi's creator Matas Kaminskas, aimed to compare the …
-
AI ranking index increases private task weight, but noise concerns remain
The Artificial Analysis Intelligence Index v4.3 has increased the weight of evaluations using private tasks or answers from 40% to 45%. This change aims to mitigate issues like gaming and contamination in AI model ranki…
-
DeepSeek-V4.1-Flash-1017 launches with 1M context, Haiku 5.5 cost guard, and LFORLA evaluation details
Several AI-related developments are emerging, including the DeepSeek-V4.1-Flash-1017 model, which offers a 1 million token context window and an MIT license, showing significant improvements on the Terminal-Bench benchm…
-
Anthropic's Claude Haiku 5.5 shows strong benchmark performance · 2 sources tracked
Anthropic's Claude Haiku 5.5 has demonstrated strong performance across several benchmarks, including OSWorld 2.1, GDPval-AA v2.1, FrontierCode 1.1, and Terminal-Bench. The model's capabilities are being highlighted in …