SWE Bench Pro
PulseAugur coverage of SWE Bench Pro — every cluster mentioning SWE Bench Pro across labs, papers, and developer communities, ranked by signal.
- instance of Claude Opus 4-8 90%
- developed Opus 4.8 90%
- instance of Opus 4.8 90%
- instance of Terminal-Bench 2.1 90%
- instance of Zhipu AI 90%
- instance of Terminal Bench 2.0 90%
- instance of Claude Fable-5 90%
- instance of cubic metre 90%
- instance of MAI-Code-1-Flash 90%
- instance of MAI-Thinking 1 90%
- used by Minimax 90%
- instance of Minimax 90%
5 day(s) with sentiment data
Anthropic's focus on 'abstention' in Opus 4.8 will drive adoption for critical coding tasks
Opus 4.8's improved ability to abstain from answering when uncertain, rather than providing incorrect information, is a critical feature for complex coding tasks. This trait, highlighted in recent evidence, could lead to increased adoption of Claude Opus for high-stakes software development where accuracy and reliability are paramount.
SWE-Bench Pro scores are rapidly increasing, with multiple models surpassing 50%
Recent evidence shows MiniMax's M3 model achieving 59% and Microsoft's MAI-Code-1-Flash achieving 51% on SWE-Bench Pro. This indicates a significant upward trend in AI coding benchmark performance, with several models now breaking the 50% barrier.
MiniMax M3 may become a leading open-source alternative for coding tasks
MiniMax's M3 model has demonstrated strong performance on SWE-Bench Pro (59%) and Terminal Bench 2 (66%), coupled with a 1M token context window. If its accessibility and performance remain competitive, it could emerge as a preferred open-source option for developers seeking advanced coding assistance, potentially challenging proprietary models.
-
GPT-6 Astra and Claude Fable 5.1 face off in benchmark showdown
A direct comparison between OpenAI's GPT-6 Astra and Anthropic's Claude Fable 5.1 reveals distinct strengths for each advanced AI model. GPT-6 Astra excels in handling massive codebases with its 1.1M context window and …
-
OpenAI flags SWE-bench Pro coding benchmark flaws; new research reframes NN training
OpenAI has identified significant reliability issues within the SWE-bench Pro coding benchmark, a tool it had previously endorsed as a replacement for the contaminated SWE-bench benchmark. This new analysis suggests tha…
-
MetaRSI-v1 advances AI self-improvement capabilities · 1 source tracked
CosmosMind, in collaboration with several universities, has introduced MetaRSI-v1, a novel meta-recursive architecture designed to improve the process of recursive self-improvement (RSI) in AI models. This new framework…
-
Claude Opus 4.8 leads GPT-5.5 on advanced coding benchmark; governance stressed
A recent comparison of leading LLMs for coding tasks reveals GPT-5.5 and Claude Opus 4.8 are nearly tied on the SWE-bench Verified benchmark, both achieving around 88.7%. However, Claude Opus 4.8 demonstrates a signific…
-
ByteDance's HarnessDev benchmark tests LLMs' ability to build agent code
Researchers from ByteDance Seed and other institutions have introduced HarnessDev, a new benchmark designed to evaluate an LLM's ability to create its own agent harnesses. Unlike traditional benchmarks that fix the harn…
-
New Agentic Coding Index ranks LLMs by intelligence density
A Reddit user has developed a new metric called the Agentic Coding Index (ACI) to evaluate Large Language Models (LLMs) on coding tasks. The ACI aggregates scores from several coding benchmarks, including SWE-bench Pro,…
-
SWE-Prime filters LLM training data for better software resolution
Researchers have introduced SWE-Prime, a novel two-stage method designed to enhance the performance of large language models in resolving software issues. This approach focuses on meticulously filtering agent trajectory…
-
Alibaba previews Qwen4 architecture with cost-efficient Qwen3.8-Flash-Next model
Alibaba's Qwen team has released Qwen3.8-Flash-Next, an open-weight multimodal MoE model that previews the architecture for the upcoming Qwen4. This new model boasts significant cost-efficiency, activating only 6B param…
-
Microsoft's AutoSaddler automates AI agent harness optimization
Researchers from Microsoft have developed AutoSaddler, a novel system designed to automatically optimize AI agent harnesses. This method treats the harness as code and learns to patch it offline using failure traces. Au…
-
Anthropic's Fable 5 struggles as businesses favor cheaper Opus 4.8
Despite developing highly capable AI models like Fable 5, Anthropic is experiencing a shift in enterprise spending towards its older, more affordable Opus 4.8 model. Data from Ramp indicates that Opus 4.8 accounts for a…
-
LLM production readiness requires 10 tests beyond benchmarks
An LLM evaluation checklist for 2026 emphasizes moving beyond simple benchmark scores to comprehensive production readiness testing. The approach, developed by Quokka Labs, focuses on evaluating the entire AI system, in…
-
Qwen3.8-27B open-source model tops leaderboards with advanced agency
Alibaba's Qwen team has open-sourced Qwen3.8-27B, a 27-billion parameter model that has achieved top rankings on several benchmarks, including SWE Bench Pro and OSWorld. This model demonstrates significant advancements …
-
九章智算云 focuses on training-inference consistency for AI infrastructure
九章智算云 is developing an AI infrastructure system focused on "training-inference consistency" to support the increasing reliance on reinforcement learning (RL) for scaling model capabilities. This system aims to efficient…
-
Alibaba's Qwen 3.8-27B model surpasses Claude Opus 4.6 Max on SWE-Bench Pro
Alibaba's new open-source model, Qwen 3.8-27B, has outperformed Anthropic's Claude Opus 4.6 Max on the SWE-Bench Pro benchmark, achieving a score of 61.7 compared to 53.4. This 27-billion parameter model is capable of r…
-
Qwen3.8-27B open-source model rivals Claude Opus on benchmarks
Qwen has released its Qwen3.8-27B model, an open-source model that rivals the performance of larger, proprietary models like Claude Opus 4.6 Max on various benchmarks, particularly in coding and agent tasks. This 27-bil…
-
Sonnet 5 achieves 63.2% on SWE-bench Pro; OpenAI's Terra claims unverified benchmark score
A new AI model, Sonnet 5, has achieved a score of 63.2% on the SWE-bench Pro benchmark. Separately, OpenAI's model, Terra, reportedly scored 84.3% on the Terminal-Bench, though this claim is vendor-stated, preview-only,…
-
CalibForge system synthesizes challenging AI agent training tasks
Researchers have developed CalibForge, a system designed to synthesize and refine terminal tasks for training AI agents. This system uses adversarial solver calibration, employing strategies like multi-solver disagreeme…
-
Tsinghua University releases VeriLoop Coder-E1 for verifiable code repair
Researchers from Tsinghua University have open-sourced VeriLoop Coder-E1, a model designed for verifiable recursive self-improvement in code repair. Built upon the Qwen3.6-27B architecture, VeriLoop Coder-E1 utilizes an…
-
xAI releases Grok 4.5 trained on real developer workflows · 1 source tracked
xAI has released Grok 4.5, a 1.5-trillion-parameter Mixture-of-Experts model trained on real developer interaction data from the Cursor IDE. This unique training approach, which includes multi-file diffs and debugger se…
-
MindForge pipeline trains small LLMs for full software engineering lifecycle
Researchers have developed MindForge, an automated pipeline designed to train smaller language models in comprehensive software engineering tasks. This system converts open-source command-line programs into source-free …