HumanEval
PulseAugur coverage of HumanEval — every cluster mentioning HumanEval across labs, papers, and developer communities, ranked by signal.
9 day(s) with sentiment data
-
LLM-generated code fails to meet developer intent, study finds
A new study published on arXiv introduces DevIntent, a benchmark designed to measure how often Large Language Models (LLMs) generate code that violates implicit developer intentions. The research found that both Claude …
-
New method retrofits linear attention to speed up diffusion language models
Researchers have developed a method to retrofit linear attention into diffusion language models (dLLMs) to accelerate inference. This new approach, called block-hybrid attention, combines exact softmax attention within …
-
LLM performance varies; task-specific capabilities matter more than rankings
A recent experiment revealed that the performance of large language models can vary significantly even when using the same tasks and parameters, challenging the notion of a single "best" model. Across two runs on 164 Hu…
-
New ABC-GRPO algorithm enhances LLM training stability and performance
Researchers have introduced All-Quadrant Bounded Clipping GRPO (ABC-GRPO), a novel algorithm designed to improve the stability and generalizability of reinforcement learning for large language models. ABC-GRPO addresses…
-
Qwen3 235B leads agentic benchmark, highlighting tool-use differences
The Agentic Index, a benchmark for multi-step task completion involving tool use and error recovery, shows a significant divergence from traditional chat leaderboards. Qwen3 235B, a Mixture-of-Experts model, has achieve…
-
AMD releases open Instella-MoE-16B LLM with 2.8B active parameters
AMD has released Instella-MoE-16B-A3B, an open-source Mixture-of-Experts language model. This model features 16 billion total parameters but only activates 2.8 billion per token, utilizing architectural innovations like…
-
Claude 3.5 Sonnet leads in AI code auditing benchmark, beating GPT-4o and Llama 3
A recent benchmark evaluated GPT-4o, Claude 3.5 Sonnet, and Llama 3 70B for their effectiveness in automated code auditing, specifically for detecting vulnerabilities in smart contracts. Claude 3.5 Sonnet emerged as the…
-
CodexGraph system enhances LLM interaction with code repositories
Researchers have developed CodexGraph, a novel system designed to improve how large language models (LLMs) interact with entire code repositories. Unlike existing methods that rely on similarity retrieval or task-specif…
-
New research questions reliability of coding agent repair loops
A new research paper titled "Looping Is Not Reliability" explores the limitations of current coding agent repair methods. The study highlights that simple repetition in generate-test-revise loops does not guarantee reli…
-
grug-27b model released, drastically cutting token usage with efficient reasoning
A new model named grug-27b, based on Qwen/Qwen3.6-27B, has been released with a focus on efficient reasoning. It utilizes a LoRA method and a novel "think-only" loss on agent trajectories, significantly reducing token u…
-
Bad Theory Labs releases BTL-3 agent model for coding tasks
Bad Theory Labs has released BTL-3, a 27-billion parameter open-weight model designed for agentic coding and structured tool use. This model is a post-trained version of Qwen3.6-27B, offering strong performance on codin…
-
Chinese Open-Source LLMs Offer Major Cost Savings Over GPT-5
Open-source Large Language Models from China are emerging as a cost-effective alternative to models like GPT-5, with some startups reporting significant savings. These models leverage efficient Mixture-of-Experts (MoE) …
-
DeepSeek, GLM, and Qwen: Chinese LLMs Compared for Free API Use
Three leading Chinese AI labs, DeepSeek, Zhipu AI (GLM), and Alibaba Cloud (Qwen), offer powerful, free LLM APIs that cater to different project needs. DeepSeek-V2, with its Mixture-of-Experts architecture, provides the…
-
AI orchestration emerges as key differentiator beyond individual models · 2 sources tracked
A new research paper introduces INFORM, an interpretability analysis tool designed to disentangle the structure and function of multi-expert Large Language Model (LLM) orchestration systems. The study, which utilized mo…
-
DeepSeek V4 Pro challenges GPT-5 and Claude 4 on benchmarks, offering superior value · 2 sources tracked
New benchmarks from mid-2026 indicate that Chinese LLM providers, particularly DeepSeek, are now competitive with or surpassing top-tier models from OpenAI and Anthropic in performance and cost-effectiveness. DeepSeek V…
-
Qwythos-9B language model benchmarked on GSM8K, IFEval, and HumanEval
A user benchmarked the Qwythos-9B language model, a fine-tune of Qwen 3.5 9B and Claude, across several standard evaluations. The model was tested on GSM8K for mathematical reasoning, IFEval for instruction following, a…
-
Alibaba-Tsinghua paper on dLLM reasoning wins ICML Outstanding Paper award
A collaborative paper from Alibaba and Tsinghua University, titled "The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models," has been recognized with an Outstanding Paper award at ICM…
-
New AI coding agent achieves frontier accuracy at 8x lower cost
A solo founder has developed a new AI coding agent that routes requests to the most cost-effective model, escalating to a frontier model only when necessary. This approach achieves parity with frontier models on the Hum…
-
LLM Benchmarking: Why Standard Evals Fail and How to Build a Better Harness
A year-long evaluation of large language models (LLMs) in production revealed that standard benchmarks like MMLU and HumanEval do not accurately predict real-world performance. The author found that exact string matchin…
-
New benchmark AlgoBench tests LLMs' algorithmic reasoning beyond memorization
Researchers have developed AlgoBench, a new framework designed to evaluate the algorithmic reasoning capabilities of code generation models. Unlike traditional benchmarks that can be compromised by training data exposur…