ProgramBench
PulseAugur coverage of ProgramBench — every cluster mentioning ProgramBench across labs, papers, and developer communities, ranked by signal.
- competes with GPT 5.6 "Sol" 80%
- instance of GPT 5.6 "Sol" 70%
- used by Terminal-Bench 4.0 70%
- used by NL2Repo-Bench 70%
- competes with organ of the Kyushu Music Kindergarten 70%
- used by GLM-5.2 70%
- competes with Kimi k3 70%
- competes with Claude (Opus 5.5) 70%
- used by Claude (Opus 5.5) 50%
- used by organ of the Kyushu Music Kindergarten 50%
10 day(s) with sentiment data
-
Spine runtime improves AI coding agents by structuring subtask context
Researchers have developed Spine, a new runtime and interface designed to manage long-horizon coding tasks by structuring subtask context. Unlike sequential history recording or recursive subagent delegation, Spine orga…
-
AI agents forget rules after first compaction, study finds
A new preprint study titled "Benchmarks Are Surprisingly Compaction-Friendly" reveals that AI coding agents lose approximately one-third of their given rules after the first compaction process. The research, conducted b…
-
AI startup Resolution autoformalizes math for safety; Claude Opus 5.5 excels on ProgramBench
AI startup Resolution has published work demonstrating the autoformalization of mathematical statements related to statistical learning theory, specifically the resolution of singularities. This achievement, which invol…
-
Context compaction impacts LLM agents, dropping constraints and instructions
A new arXiv paper explores the impact of context compaction on large language model agents, particularly in long-horizon tasks that exceed their context windows. Researchers found that while compaction strategies did no…
-
Microsoft Research open-sources Agensh for 1,024 AI agents
Microsoft Research has open-sourced Agensh, a new multi-agent framework designed to coordinate large numbers of AI agents without a central orchestrator. This decentralized approach allows individual agents to pick thei…
-
Kimi K3.1 Pro leads LLM front-end benchmarks amid Qwen 4, Claude Fable 5.5 uncertainty · 1 source tracked
A comparison of three large language models—Kimi K3.1 Pro, Qwen 4, and Claude Fable 5.5—highlights Kimi K3.1 Pro's early availability and competitive pricing for front-end development tasks. While Moonshot has released …
-
HarnessPro improves AI agent performance and cuts costs
Researchers have developed HarnessPro, a novel method for improving AI agent harnesses by recursively enhancing their performance while simultaneously reducing execution costs. This label-free approach utilizes LLM-gene…
-
DeepSeek V4.1 Flash 1017 model shows strong performance on coding benchmarks
DeepSeek has released V4.1 Flash 1017, an open-source model with a 1M token context window and 763B parameters. This iteration shows significant improvements across eleven benchmarks, particularly in coding tasks where …
-
DeepSeek V4.1 Flash 1017 debuts with major gains in coding and agent tasks
DeepSeek has released an updated version of its open-source large language model, DeepSeek V4.1 Flash 1017, on October 17, 2026. This iteration features a significant improvement across various benchmarks, particularly …
-
HarnessPro method improves AI agent performance and cuts costs
A new method called HarnessPro has been developed to recursively improve AI agent harnesses, aiming to enhance task performance while simultaneously reducing execution costs. This label-free approach utilizes LLM-genera…
-
DeepSeek releases V4.1-Flash-1017 with improved coding performance and lower costs · 1 source tracked
DeepSeek has released an updated version of its V4.1-Flash model, named DeepSeek V4.1-Flash-1017. This new open-source model, available under the MIT license on Hugging Face, claims significant improvements in coding be…
-
Meta's Muse Wonder AI models launch amid mixed reviews and trust concerns
Meta Superintelligence Labs has released its first frontier-class AI models, Muse Wonder and Muse Wonder Full, achieving competitive scores on various benchmarks, including the Artificial Analysis Intelligence Index whe…
-
Meta's Muse Wonder AI challenges Anthropic's Opus 5.5 in coding benchmarks
Meta has launched Muse Wonder, a new AI model that competes with Anthropic's Claude Opus 5.5, particularly in long coding tasks. While Meta's internal benchmarks show Muse Wonder outperforming Opus 5.5 in certain areas …
-
DeepSeek V4.1 Flash launches, slashing agentic coding costs by 36x
DeepSeek V4.1 Flash, a new large language model, has been released with a significantly lower cost for specific workloads, costing approximately 36 times less than Anthropic's Claude Opus 5 for agentic coding tasks. Des…
-
New framework enables LLM agents to autonomously develop and improve software
Researchers have introduced Harness-of-Harness (HoH), a framework designed to enhance autonomous software development by enabling LLM-based coding agents to continually improve software. HoH structures development into …
-
Anthropic's Opus 5 sets new SoTA on ProgramBench, beating GPT Sol
Anthropic's Opus 5 model has achieved a new state-of-the-art performance on the ProgramBench benchmark, successfully solving 9 out of 200 task instances. This represents a more than fourfold improvement compared to the …
-
MindForge pipeline trains small LLMs for full software engineering lifecycle
Researchers have developed MindForge, an automated pipeline designed to train smaller language models in comprehensive software engineering tasks. This system converts open-source command-line programs into source-free …
-
Moonshot AI releases open-weight Kimi K3 for complex coding tasks · 2 sources tracked
Moonshot AI has released Kimi K3, an open-weight model with 2.8 trillion parameters designed for long-horizon coding and agentic tasks. The model features a one-million-token context window and a Mixture-of-Experts arch…
-
Fable 5 benchmark shows double Opus 4.8 performance
A new benchmark, ProgramBench, has been used to evaluate Fable 5, with results suggesting it significantly outperforms Opus 4.8. The benchmark creator noted that Fable 5's performance was double that of Opus 4.8, even w…
-
ProgramBench coding benchmark fails frontier models due to impossible undocumented tests
A new coding benchmark called ProgramBench, designed to evaluate frontier AI models, has been criticized for being potentially impossible to solve. The benchmark requires models to reimplement programs based on limited …