SWE-bench
PulseAugur coverage of SWE-bench — every cluster mentioning SWE-bench across labs, papers, and developer communities, ranked by signal.
6 天有情绪数据
-
AgentEval framework improves AI agent workflow evaluation with DAG-based error tracking
Researchers have developed AgentEval, a new framework for evaluating agentic workflows by representing them as directed acyclic graphs (DAGs). This approach allows for detailed step-level assessment and tracking of erro…
-
SWE-bench tests AI agents' real-world capability, showing 80% resolution rate
Evaluating the real-world performance of AI agents is becoming critical as they transition from experimental stages to production environments. Traditional metrics like perplexity scores are insufficient for assessing a…
-
Public AI models replicate Anthropic's vulnerability discovery findings
Researchers have successfully replicated Anthropic's Mythos findings using publicly available AI models like GPT-5.4 and Claude Opus 4.6. This suggests that advanced AI capabilities for discovering software vulnerabilit…
-
How good are LLMs at fixing their mistakes? A chatbot arena experiment with Keras and TPUs
Current methods for evaluating large language models, such as MMLU and HumanEval, may be insufficient as they do not capture the nuances of interactive, goal-oriented conversations. A more effective approach would invol…
-
Cosine Genie leverages GPT-4o fine-tuning to become top coding agent
Cosine has launched Genie, a coding agent that has achieved the top ranking on the SWE-Bench benchmark, surpassing previous leaders by a significant margin. This success is attributed to fine-tuning OpenAI's GPT-4o mode…
-
OpenAI abandons SWE-bench Verified due to flawed tests and data contamination
OpenAI has announced it will no longer use SWE-bench Verified to evaluate the coding capabilities of frontier AI models. The benchmark has become contaminated, with models showing improved scores primarily due to exposu…
-
OpenAI releases GPT-4o with fine-tuning and enhanced multimodal capabilities
OpenAI has released fine-tuning capabilities for its GPT-4o model, allowing developers to customize its performance and tone for specific applications. This feature, available on paid tiers, offers developers the chance…
-
Introducing gpt-realtime and Realtime API updates
OpenAI has released GPT-4.1, a new series of models for its API that offer significant improvements in coding, instruction following, and long context comprehension, outperforming previous models like GPT-4o. The compan…