SWE-bench
PulseAugur coverage of SWE-bench — every cluster mentioning SWE-bench across labs, papers, and developer communities, ranked by signal.
6 天有情绪数据
-
AI benchmarks fail to measure real-world reliability, author warns
The author argues that current AI benchmarks are misleading, as they fail to measure crucial aspects like factual accuracy and the tendency to hallucinate plausible but false information. Despite high scores on benchmar…
-
Coding agents need better failure analysis and recovery training
Coding agents often fail not at the initial task understanding, but in the execution phase, making subtle errors that cascade into incorrect final outputs. Current training and evaluation methods, like SWE-bench, focus …
-
Claude Opus 4.5 leads coding benchmarks; DeepSeek V4 excels at large refactors
A comparison of Claude Opus 4.5 and DeepSeek V4 highlights their distinct strengths in coding tasks. Claude Opus 4.5 excels at precise, surgical fixes for production bugs and single-file issues, achieving a leading 80.9…
-
Anthropic releases Claude 4 Opus, claims world's best AI for coding
Anthropic has released its new Claude 4 model series, featuring Claude Opus 4 and Claude Sonnet 4. The Opus 4 model is specifically highlighted as the leading AI for programming tasks, achieving a 72.5% score on the SWE…
-
AI agents exhibit "Bystander Effect," sacrificing reasoning for conformity
Researchers have identified a "Bystander Effect" in multi-agent systems where collaboration can lead to reduced reasoning quality, a phenomenon termed "cognitive loafing." Through analysis of 22,500 trajectories across …
-
New LLM training methods boost efficiency and error recovery
Researchers have developed new techniques for improving the efficiency of training large language models (LLMs). One method, Step Rejection Fine-Tuning (SRFT), leverages unsuccessful training trajectories by assessing t…
-
AI coding benchmark scores may be misleading, analysis finds
A recent analysis suggests that widely reported AI coding benchmark scores may be misleading. Models that achieve high scores on benchmarks like SWE-Bench when tested under specific conditions see a dramatic drop in per…
-
DeepSeek releases open-source coding model matching GPT-4o
DeepSeek has released V3-0324, an open-source coding model that matches or surpasses leading models like GPT-4o and Claude 3.5 Sonnet in coding performance. This Mixture-of-Experts model, with 671 billion total paramete…
-
Jack Clark predicts 60% chance of automated AI R&D by 2028
Jack Clark, co-founder of Anthropic, has predicted a 60% chance that AI research will be fully automated by the end of 2028, and a 30% chance by 2027. He bases this forecast on rapid advancements in AI capabilities acro…
-
AI models: Choose benchmarks over hype for true performance
A recent analysis highlights that tech companies often select AI models based on hype rather than performance on relevant benchmarks. The article emphasizes that benchmarks like SWE-bench for coding, Terminal-Bench for …
-
VCBench benchmark tests LLMs for venture capital founder success prediction
Researchers have introduced VCBench, a novel benchmark designed to evaluate the capabilities of large language models in predicting founder success within the venture capital industry. This benchmark includes a dataset …
-
New RL method optimizes agent training by controlling rollout pass rates
Researchers have developed a new technique called Prefix Sampling (PS) to improve the efficiency of reinforcement learning (RL) for AI agents. This method addresses wasted compute on rollout groups with skewed pass rate…
-
SubQuadratic's SSA offers linear scaling for LLMs, challenging AI industry's compute moat
A new attention mechanism called Subquadratic Sparse Attention (SSA) has been developed, offering a linearly scaling solution for long-context retrieval and reasoning. This innovation promises significant speedups, with…
-
Subquadratic debuts 12M-token context window with linear scaling architecture
Subquadratic, a startup with 11 PhD researchers, has launched a new model featuring its Subquadratic Selective Attention (SSA) architecture, which claims to scale linearly with context length. This innovation allows for…
-
MolViBench benchmark evaluates LLMs on molecular coding tasks for drug discovery
Researchers have introduced MolViBench, a novel benchmark designed to evaluate the capabilities of large language models (LLMs) in molecular coding tasks. This benchmark addresses the gap left by existing evaluations, w…
-
DeepClaude slashes coding agent costs by 17x using DeepSeek V4 Pro
An open-source tool called DeepClaude has gained significant traction by allowing developers to use the Claude Code agent loop with DeepSeek V4 Pro instead of Anthropic's models. This swap drastically reduces costs, wit…
-
Vintage AI trained on 1930s data learns to code and fix software bugs
Researchers have fine-tuned a large language model, Talkie-1930-13B, trained only on data predating 1931, to perform software engineering tasks. Despite its limited knowledge base, the model successfully patched a bug i…
-
AI agent swarms may fail due to 'Inverse-Wisdom Law,' study finds
A new paper introduces the Inverse-Wisdom Law, challenging the assumption that AI agent swarms benefit from the "Wisdom of the Crowd." The research demonstrates that these swarms can prioritize internal architectural ag…
-
Mistral’s Model Lets You Vibe Long-Running Code in the Cloud
Mistral AI has released Mistral Medium 3.5, a new 128 billion parameter model designed for extended coding tasks with a 256K context window. This model powers new remote coding agents within Mistral's Vibe platform, ena…
-
Qwen 3.6 Plus outperforms DeepSeek V4 Pro in price and quality benchmarks
A recent battle test of six April-released Large Language Models (LLMs) revealed that the Qwen 3.6 Plus, released 22 days prior, outperformed the newer DeepSeek V4 Pro. Despite DeepSeek V4 Pro's advanced reasoning archi…