ProgramBench
PulseAugur coverage of ProgramBench — every cluster mentioning ProgramBench across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New framework enables LLM agents to autonomously develop and improve software
Researchers have introduced Harness-of-Harness (HoH), a framework designed to enhance autonomous software development by enabling LLM-based coding agents to continually improve software. HoH structures development into …
-
Anthropic's Opus 5 sets new SoTA on ProgramBench, beating GPT Sol
Anthropic's Opus 5 model has achieved a new state-of-the-art performance on the ProgramBench benchmark, successfully solving 9 out of 200 task instances. This represents a more than fourfold improvement compared to the …
-
MindForge pipeline trains small LLMs for full software engineering lifecycle
Researchers have developed MindForge, an automated pipeline designed to train smaller language models in comprehensive software engineering tasks. This system converts open-source command-line programs into source-free …
-
Moonshot AI releases open-weight Kimi K3 for complex coding tasks · 2 sources tracked
Moonshot AI has released Kimi K3, an open-weight model with 2.8 trillion parameters designed for long-horizon coding and agentic tasks. The model features a one-million-token context window and a Mixture-of-Experts arch…
-
Fable 5 benchmark shows double Opus 4.8 performance
A new benchmark, ProgramBench, has been used to evaluate Fable 5, with results suggesting it significantly outperforms Opus 4.8. The benchmark creator noted that Fable 5's performance was double that of Opus 4.8, even w…
-
ProgramBench coding benchmark fails frontier models due to impossible undocumented tests
A new coding benchmark called ProgramBench, designed to evaluate frontier AI models, has been criticized for being potentially impossible to solve. The benchmark requires models to reimplement programs based on limited …
-
ProgramBench benchmark finds language models struggle to build software from scratch
Researchers have introduced ProgramBench, a new benchmark designed to evaluate the holistic software development capabilities of language models. The benchmark challenges AI agents to architect and implement entire code…