SWE-bench
PulseAugur coverage of SWE-bench — every cluster mentioning SWE-bench across labs, papers, and developer communities, ranked by signal.
- authored by arXiv 90%
- instance of Qwen3.5:9b 90%
- instance of Claude Sonnet 4.6 90%
- used by large-language models 90%
- used by LiveCodeBench 90%
- instance of DeepSeek V4-Pro 90%
- used by arXiv 70%
- used by ScienceCast 70%
- used by alphaXiv 70%
- used by DagsHub 70%
- used by GLM-5.2 70%
- instance of HumanEval 70%
18 day(s) with sentiment data
-
AI model evaluation evolves from token prediction to complex workflows
The evaluation of frontier AI models has evolved through distinct eras, moving from simple token prediction and log-likelihood grading to assessing instruction following, then to verifying math, code, factuality, and sa…
-
Debugger vs. Bash: AI Coding Agents Show Bash Outperforms Debuggers on SWE-bench
A new study compares the effectiveness of interactive debuggers versus traditional edit-execute-observe loops for AI coding agents. Researchers evaluated two models, GPT-OSS-120B and Nemotron-3-Ultra-550B, on the SWE-be…
-
Liquid AI unveils efficient 1GB LFM3 model family for local agents
Liquid AI has released LFM3, a new family of open-weight models designed for efficiency and local deployment. The flagship LFM3-1.7B model, weighing approximately 1 GB, boasts a 256K-token context window and a unique tr…
-
LLM agents improve with detailed tool descriptions, not just system prompts
Developers can improve LLM agent performance by meticulously crafting tool descriptions, as these descriptions function as prompts that the models interpret. Anthropic's research indicates that refining tool definitions…
-
Coding agents suffer catastrophic forgetting during harness evolution
A new paper investigates the challenge of "catastrophic forgetting" in coding agents that undergo continual harness self-evolution. Researchers found that as these agents evolve to handle new tasks, their performance on…
-
AI agents often fail to detect early mistakes in long tasks, study finds
A recent study by researchers from UCLA, Google Research, Google DeepMind, and NYU investigated the reliability of AI agents on long-horizon tasks. By analyzing over 2,500 agent trajectories across software engineering,…
-
AI system ExLabel enhances software engineering dataset quality
Researchers have developed ExLabel, an AI-powered system designed to improve the quality of datasets used for training and evaluating large language models (LLMs) in software engineering tasks. ExLabel uses an evolving …
-
Microsoft and UC Berkeley unveil TokenEconomizer to cut coding agent costs
Researchers from Microsoft Research and UC Berkeley have developed TokenEconomizer, an open-source Python package designed to reduce the costs associated with coding agents. This plugin learns from past interactions wit…
-
Aetox harness optimizes LLM performance and token usage on Windows
The developer of the Aetox harness has detailed its performance on Windows, highlighting how its architecture impacts LLM results. The harness optimizes token usage by mapping knowledge and only fetching model bodies wh…
-
AI Skill Factory Solves Multi-Agent Drift and Context Issues
A development team has created an open-source tool to address common structural failures in AI agent skills. The tool, named the AI Skill & DevTool Commercialization Factory, aims to prevent issues like multi-agent targ…
-
Moonshot AI releases Kimi K3.1 Pro, deliberately downgrading cyber capabilities for safety
Moonshot AI has released Kimi K3.1 Pro, a large open-weight model with a 1-million-token context window. Notably, the company deliberately removed cyber-offense training data from this model, resulting in a measurable d…
-
AI evaluation firm Authensor shifts focus to auditing AI graders, fixing bugs
Authensor, an AI evaluation integrity firm, has retired its SafeClaw and Authensor policy engine products to focus on auditing AI grading tools. The company has identified and fixed numerous bugs across various AI evalu…
-
Software engineering principles enhance coding LLMs, paper shows
A new paper explores how software engineering principles can be applied to enhance the development and capabilities of large language models (LLMs) designed for coding tasks. Researchers developed a pipeline called Refa…
-
New research quantifies physical costs of context in coding AI agents
A new research paper from arXiv explores the physical costs associated with context management in coding agents. The study introduces a quality-gated protocol to measure the energy and time expenditure of these agents, …
-
New AI method SWE-Ladder enhances software repair capabilities
Researchers have developed SWE-Ladder, a novel approach to training AI agents for software engineering tasks. This method utilizes progressive privileged self-distillation, breaking down the repair process into distinct…
-
AI agent restart monitors evaluated by decision value, not just accuracy
A new research paper proposes a decision-value framework for evaluating AI agent restart monitors, arguing that traditional metrics like accuracy are insufficient. The framework suggests that the value of restarting an …
-
GEO and SEO are distinct metrics; agent recipes improve AI reliability
A new approach to Generative Engine Optimization (GEO) distinguishes it from traditional Search Engine Optimization (SEO), arguing that GEO scores measure content's quotability and machine-parseability, while SEO focuse…
-
Tsinghua University open-sources AKV for efficient LLM agent KV cache management · 4 sources tracked
Researchers from Tsinghua University and The Chinese University of Hong Kong have open-sourced AKV, a novel serving system designed for KV cache engineering in LLM agents. Unlike traditional context engineering that sum…
-
New benchmark reveals context compaction costs for AI coding agents
A new benchmark called Compact-Bench has been developed to evaluate the impact of context compaction on long-horizon coding agents. The benchmark pairs full-history and summary-only continuations to measure how context …
-
Phronix paradigm enables AI agents to learn and reuse capabilities
Researchers have introduced Phronix, a novel capability-centric learning paradigm designed to enhance the performance of AI agents in scientific computing. This system operates on a predictor-validator-corrector loop, e…