pass@k
PulseAugur coverage of pass@k — every cluster mentioning pass@k across labs, papers, and developer communities, ranked by signal.
8 day(s) with sentiment data
-
New Seq@K metric evaluates language models on iterative feedback
A new evaluation protocol called Seq@K has been introduced to better assess language models' ability to learn from sequential feedback. Unlike traditional metrics like Pass@1 or Pass@K, Seq@K measures a model's success …
-
RLVR offers no free lunch: gains in pass@1 incur pass@k costs, study finds · 3 sources tracked
A new theoretical paper argues that Reinforcement Learning with Verifiable Rewards (RLVR) does not fundamentally improve a language model's reasoning capabilities. Instead, it demonstrates that any gain in solving probl…
-
Claude Code evaluation metrics detailed for AI agent consistency
This article details three key metrics for evaluating AI agents, specifically focusing on Claude Code. The metrics are pass@k, which checks if an agent can solve a task within a set number of attempts; pass^k, measuring…
-
New paper details comprehensive framework for large model training stages
A new paper on arXiv provides a comprehensive framework for understanding and optimizing the entire lifecycle of large model training. It proposes a six-dimensional analytical framework covering data, objective function…
-
Outcome RL models boost performance via finishing and reliability, not new knowledge
A new research paper, "What Outcome RL Buys: Finishing, Reliability, Rarely Reach," by Yue et al. investigates the effectiveness of Outcome Reinforcement Learning (RL) models. The study compares 20 released RL models ag…
-
New pruning method optimizes reasoning models' pass@k metric
Researchers have developed a new pruning procedure for reasoning models that directly optimizes the pass@k metric. This method balances the quality of candidate solutions with their diversity, aiming to maximize the pro…
-
New arXiv Paper Distinguishes Agent Harness Optimization Gains
A new paper from arXiv explores the distinction between elicitation and expansion in agent harness optimization for language models. The research analyzes whether optimized harnesses improve performance on tasks already…
-
LLM research reveals prompt interference degrades single-shot performance
A new research paper explores the phenomenon of "prompt interference" in large language models, where optimizing for the pass@k metric can inadvertently degrade performance on the pass@1 metric. This occurs because shar…
-
AI model evaluation metric pass@k questioned for diversity and capability retention
A new research paper questions the effectiveness of the pass@k metric in evaluating AI models, particularly after post-training fine-tuning. The study demonstrates that while pass@k might show marginal improvements, it …
-
Diffusion models research tackles coverage, tabular data, and efficiency · 6 sources tracked
Recent research explores advancements in diffusion models, focusing on improving their efficiency and coverage. One paper introduces 'pass@k' to evaluate distribution coverage in distilled diffusion models, revealing th…
-
New research highlights limitations in AI evaluation methods
A new paper explores the limitations of pass@k evaluations in machine learning, particularly when extrapolating beyond the number of samples collected. The research demonstrates that fixed-n success counts in conditiona…
-
New DATPO method enhances reasoning coverage in Large Reasoning Models
Researchers have developed DATPO, a new method to improve the reasoning capabilities of Large Reasoning Models trained with Reinforcement Learning with Verifiable Rewards (RLVR). DATPO addresses the limitation of RLVR i…
-
New Method Boosts Natural Language to Logic Translation Accuracy
Researchers have developed a new method called Stratified Consistency Distillation to improve the accuracy of translating natural language into logical formulas. This approach uses a frontier LLM to generate multiple lo…
-
New RL method GAPO boosts Qwen and Llama performance on benchmarks
Researchers have introduced Group Adaptive Clipping Policy Optimization (GAPO), a novel method designed to enhance reinforcement learning with verifiable rewards. GAPO adaptively adjusts clipping thresholds based on rol…
-
New RLVR method boosts LLM reasoning diversity and performance
Researchers have developed a new method to enhance the reasoning capabilities of large language models (LLMs) while maintaining their diversity. The approach, termed Reinforcement Learning with Verifiable Rewards (RLVR)…
-
AI sampling temperature theory explains budget-dependent optimization
Researchers have developed a theoretical framework to explain why the optimal sampling temperature for AI models often increases with a larger sampling budget. This phenomenon, observed across various models like Codex,…
-
New research identifies pass@k inversion in RLVR, proposes mitigation strategy
A new research paper explores the phenomenon of "pass@k inversion" in reinforcement learning with verifiable rewards (RLVR). This occurs when RLVR improves a model's one-sample accuracy but degrades its performance on t…
-
New research tackles AI code generation evaluation and testing
Two new research papers explore advancements in evaluating AI-generated code. The first, TENET, introduces a framework for repository-level code generation using test-driven development, achieving high Pass@1 scores on …
-
New research questions LLM fine-tuning effectiveness for Dart code decompilation
A new research paper explores the effectiveness of fine-tuning large language models for the neural decompilation of Dart Ahead-of-Time (AOT) binaries. The study found that fine-tuning did not significantly improve pass…
-
New SR-PPO method improves RL for language models with single rollout
Researchers have developed a new method called Single-Rollout Proximal Policy Optimization (SR-PPO) to address the challenges of estimating token-level advantages in reinforcement learning for language models. This appr…