SWE-bench Verified
PulseAugur coverage of SWE-bench Verified — every cluster mentioning SWE-bench Verified across labs, papers, and developer communities, ranked by signal.
- instance of SWE Bench Pro 90%
- instance of Apache Software License 2.0 90%
- instance of Terminal-Bench 90%
- instance of LiveCodeBench V6 90%
- developed by Flintshire 90%
- instance of AppWorld 90%
- developed by recursion 90%
- instance of Terminal-Bench V2 90%
- instance of Gotit.pub 70%
- instance of alphaXiv 70%
- instance of DagsHub 70%
- instance of Terminal-Bench 2.1 70%
17 day(s) with sentiment data
-
AI agent benchmarks flawed, new research paper reveals
A new research paper highlights critical flaws in how AI agents are benchmarked, suggesting that current evaluation methods often fail to accurately rank models before tasks are exhausted. The study re-evaluated eight b…
-
Thinking Machines releases 594B Inkling 2 Small, plans open weights
Thinking Machines has released Inkling 2 Small, a 594 billion parameter model that achieves frontier-level performance on benchmarks. The model utilizes a mixture-of-experts architecture, with only 17 billion parameters…
-
Hugging Face SWE-bench board shows 9 self-reported results above official best · 1 source tracked
The Hugging Face SWE-bench Verified board, updated on November 5, 2026, lists 78 results from 68 repositories, none of which are officially verified. Nine self-reported entries surpass the highest score of 79.2 on the s…
-
NVIDIA paper shows aggressive context compaction maintains LLM accuracy with less tokens
A new paper from NVIDIA demonstrates that aggressive context compaction techniques can significantly reduce the token usage of large language models without substantial accuracy loss. The research shows that models like…
-
DeepSeek-V4.1-Flash-1017 compressed to 1-bit with expert pruning
ISTA-DASLab has released a specialized version of the DeepSeek-V4.1-Flash-1017 model, named GSQ-RCO Coder. This model is a 763B parameter mixture-of-experts architecture that has been significantly compressed by removin…
-
Hippo memory framework enables AI agent self-evolution without parameter updates
Researchers have developed Hippo, a novel memory framework for AI agents that enables self-evolution without parameter updates. Inspired by the human hippocampus, Hippo compares agent attempts at the same task to identi…
-
Amazon builds Sandman for agentic RL sandboxes on Kubernetes
Amazon has developed Sandman, a service designed to manage sandboxes for agentic reinforcement learning workloads on shared Kubernetes clusters. This system allows for the creation, operation, and cleanup of thousands o…
-
AI system ExLabel enhances software engineering dataset quality
Researchers have developed ExLabel, an AI-powered system designed to improve the quality of datasets used for training and evaluating large language models (LLMs) in software engineering tasks. ExLabel uses an evolving …
-
New benchmarks and research push AI coding agents toward complex, long-horizon tasks
Several new research papers and benchmarks are being released to evaluate the capabilities of AI coding agents, particularly in handling long-horizon tasks and complex software engineering workflows. These efforts aim t…
-
Software engineering principles enhance coding LLMs, paper shows
A new paper explores how software engineering principles can be applied to enhance the development and capabilities of large language models (LLMs) designed for coding tasks. Researchers developed a pipeline called Refa…
-
AI coding models Flint, Gemma 4, and Qwen3.8 compared on benchmarks
A comparison of three AI coding models—Flint, Gemma 4, and Qwen3.8—reveals distinct design goals and performance metrics. Flint, developed by Recursive, is optimized for code exploration and agentic tasks, boasting high…
-
Population-Centered Evolution enhances AI agent development
Researchers have introduced a new principle for AI agent evolution called Population-Centered Evolution (PCE). This method focuses on improving agents by analyzing population-wide patterns of success, failure, and self-…
-
CuraLoc plans code inspections to improve fault localization efficiency
Researchers have developed CuraLoc, a novel method for repository-level fault localization that plans code inspections before source retrieval. This approach contrasts with existing methods that use iterative model call…
-
New method distills transferable coding skills for AI agents
Researchers have developed a new method called Hierarchical Skill Evolution (HSE) to distill transferable procedural knowledge for coding agents. HSE progressively refines skills by aggregating evidence from individual …
-
New ShutdownMod method tests AI agent compliance with shutdown requests
Researchers have developed a new method called ShutdownMod that can evaluate an AI agent's compliance with shutdown requests. This technique transforms any agent's interaction history into a shutdown evaluation by inser…
-
Small coding agents' use of code graphs and embeddings analyzed
A new research paper investigates how small coding agents utilize code graphs and embedding search tools, finding that these structured tools can significantly improve accuracy for models with fewer parameters. The stud…
-
Study questions effectiveness of AI coding agent self-evolution
A new study published on arXiv explores the self-evolution capabilities of coding agents, specifically focusing on a skill called \retro. This skill allows agents to propose changes to their operational harnesses based …
-
OpenSWE framework synthesizes 45k SWE environments for agent training
Researchers have developed OpenSWE, a large-scale, transparent framework for training software engineering (SWE) agents. This framework includes over 45,000 executable Docker environments derived from more than 12,000 r…
-
Coding agent harnesses show minimal score impact, but costs vary widely · 1 source tracked
A recent study published on arXiv and covered by VentureBeat investigated the impact of different coding agent harnesses on LLM performance. Researchers from Shandong University found that when using the same underlying…
-
AI agent verification strategies improve accuracy with minimal cost increase
A new research paper explores verification strategies for AI agent harnesses, which enable large language models to perform tasks through tools and iterative refinement. The study compares 17 different verification desi…