SWE-bench
PulseAugur coverage of SWE-bench — every cluster mentioning SWE-bench across labs, papers, and developer communities, ranked by signal.
- instance of DeepSeek V4-Pro 90%
- instance of Terminal-Bench 70%
- instance of Massive Multitask Language Understanding 70%
- instance of Generative Ai Interactive Agents 70%
- acquired by SWE-bench Verified 70%
- competes with HumanEval 60%
- instance of arXiv 60%
- instance of HumanEval 60%
- authored by arXiv 50%
13 day(s) with sentiment data
-
AI models: Choose benchmarks over hype for true performance
A recent analysis highlights that tech companies often select AI models based on hype rather than performance on relevant benchmarks. The article emphasizes that benchmarks like SWE-bench for coding, Terminal-Bench for …
-
VCBench benchmark tests LLMs for venture capital founder success prediction
Researchers have introduced VCBench, a novel benchmark designed to evaluate the capabilities of large language models in predicting founder success within the venture capital industry. This benchmark includes a dataset …
-
New RL method optimizes agent training by controlling rollout pass rates
Researchers have developed a new technique called Prefix Sampling (PS) to improve the efficiency of reinforcement learning (RL) for AI agents. This method addresses wasted compute on rollout groups with skewed pass rate…
-
SubQuadratic's SSA offers linear scaling for LLMs, challenging AI industry's compute moat
A new attention mechanism called Subquadratic Sparse Attention (SSA) has been developed, offering a linearly scaling solution for long-context retrieval and reasoning. This innovation promises significant speedups, with…
-
Subquadratic debuts 12M-token context window with linear scaling architecture
Subquadratic, a startup with 11 PhD researchers, has launched a new model featuring its Subquadratic Selective Attention (SSA) architecture, which claims to scale linearly with context length. This innovation allows for…
-
MolViBench benchmark evaluates LLMs on molecular coding tasks for drug discovery
Researchers have introduced MolViBench, a novel benchmark designed to evaluate the capabilities of large language models (LLMs) in molecular coding tasks. This benchmark addresses the gap left by existing evaluations, w…
-
DeepClaude slashes coding agent costs by 17x using DeepSeek V4 Pro
An open-source tool called DeepClaude has gained significant traction by allowing developers to use the Claude Code agent loop with DeepSeek V4 Pro instead of Anthropic's models. This swap drastically reduces costs, wit…
-
Vintage AI trained on 1930s data learns to code and fix software bugs
Researchers have fine-tuned a large language model, Talkie-1930-13B, trained only on data predating 1931, to perform software engineering tasks. Despite its limited knowledge base, the model successfully patched a bug i…
-
AI agent swarms may fail due to 'Inverse-Wisdom Law,' study finds
A new paper introduces the Inverse-Wisdom Law, challenging the assumption that AI agent swarms benefit from the "Wisdom of the Crowd." The research demonstrates that these swarms can prioritize internal architectural ag…
-
Mistral’s Model Lets You Vibe Long-Running Code in the Cloud
Mistral AI has released Mistral Medium 3.5, a new 128 billion parameter model designed for extended coding tasks with a 256K context window. This model powers new remote coding agents within Mistral's Vibe platform, ena…
-
Qwen 3.6 Plus outperforms DeepSeek V4 Pro in price and quality benchmarks
A recent battle test of six April-released Large Language Models (LLMs) revealed that the Qwen 3.6 Plus, released 22 days prior, outperformed the newer DeepSeek V4 Pro. Despite DeepSeek V4 Pro's advanced reasoning archi…
-
AgentEval framework improves AI agent workflow evaluation with DAG-based error tracking
Researchers have developed AgentEval, a new framework for evaluating agentic workflows by representing them as directed acyclic graphs (DAGs). This approach allows for detailed step-level assessment and tracking of erro…
-
SWE-bench tests AI agents' real-world capability, showing 80% resolution rate
Evaluating the real-world performance of AI agents is becoming critical as they transition from experimental stages to production environments. Traditional metrics like perplexity scores are insufficient for assessing a…
-
Public AI models replicate Anthropic's vulnerability discovery findings
Researchers have successfully replicated Anthropic's Mythos findings using publicly available AI models like GPT-5.4 and Claude Opus 4.6. This suggests that advanced AI capabilities for discovering software vulnerabilit…
-
How good are LLMs at fixing their mistakes? A chatbot arena experiment with Keras and TPUs
Current methods for evaluating large language models, such as MMLU and HumanEval, may be insufficient as they do not capture the nuances of interactive, goal-oriented conversations. A more effective approach would invol…
-
Cosine Genie leverages GPT-4o fine-tuning to become top coding agent
Cosine has launched Genie, a coding agent that has achieved the top ranking on the SWE-Bench benchmark, surpassing previous leaders by a significant margin. This success is attributed to fine-tuning OpenAI's GPT-4o mode…
-
OpenAI abandons SWE-bench Verified due to flawed tests and data contamination
OpenAI has announced it will no longer use SWE-bench Verified to evaluate the coding capabilities of frontier AI models. The benchmark has become contaminated, with models showing improved scores primarily due to exposu…
-
OpenAI releases GPT-4o with fine-tuning and enhanced multimodal capabilities
OpenAI has released fine-tuning capabilities for its GPT-4o model, allowing developers to customize its performance and tone for specific applications. This feature, available on paid tiers, offers developers the chance…
-
Introducing gpt-realtime and Realtime API updates
OpenAI has released GPT-4.1, a new series of models for its API that offer significant improvements in coding, instruction following, and long context comprehension, outperforming previous models like GPT-4o. The compan…