Pass@1
PulseAugur coverage of Pass@1 — every cluster mentioning Pass@1 across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
-
New Seq@K metric evaluates language models on iterative feedback
A new evaluation protocol called Seq@K has been introduced to better assess language models' ability to learn from sequential feedback. Unlike traditional metrics like Pass@1 or Pass@K, Seq@K measures a model's success …
-
RLVR offers no free lunch: gains in pass@1 incur pass@k costs, study finds · 3 sources tracked
A new theoretical paper argues that Reinforcement Learning with Verifiable Rewards (RLVR) does not fundamentally improve a language model's reasoning capabilities. Instead, it demonstrates that any gain in solving probl…
-
New paper details comprehensive framework for large model training stages
A new paper on arXiv provides a comprehensive framework for understanding and optimizing the entire lifecycle of large model training. It proposes a six-dimensional analytical framework covering data, objective function…
-
Outcome RL models boost performance via finishing and reliability, not new knowledge
A new research paper, "What Outcome RL Buys: Finishing, Reliability, Rarely Reach," by Yue et al. investigates the effectiveness of Outcome Reinforcement Learning (RL) models. The study compares 20 released RL models ag…
-
New arXiv Paper Distinguishes Agent Harness Optimization Gains
A new paper from arXiv explores the distinction between elicitation and expansion in agent harness optimization for language models. The research analyzes whether optimized harnesses improve performance on tasks already…
-
LLM research reveals prompt interference degrades single-shot performance
A new research paper explores the phenomenon of "prompt interference" in large language models, where optimizing for the pass@k metric can inadvertently degrade performance on the pass@1 metric. This occurs because shar…
-
AI research reveals counterintuitive findings on observation fidelity, consciousness, and reasoning reliability
New research indicates that increasing the fidelity of observations can paradoxically harm the problem-solving abilities of embodied Large Language Models (LLMs), with one study finding raw RGB input to be more effectiv…
-
Study: LLM-generated comments boost code generation if they contain correct solutions
A new study published on arXiv investigates how natural language comments generated by large language models (LLMs) impact code generation performance. Researchers found that comments derived from successful code soluti…
-
New LLM Framework Integrates Reasoning and Self-Critique
Researchers have developed a new framework called Stepwise Think-Critique (STC) that enables a single large language model to perform interleaved reasoning and self-critique. Unlike existing models that separate these p…
-
DART framework uses DAG and blockchain for trustworthy LLM multi-agent collaboration
Researchers have introduced DART, a new framework designed to enhance trust and accountability in large language model (LLM) multi-agent systems. DART utilizes a Directed Acyclic Graph (DAG) structure for workflow orche…
-
New RL method GAPO boosts Qwen and Llama performance on benchmarks
Researchers have introduced Group Adaptive Clipping Policy Optimization (GAPO), a novel method designed to enhance reinforcement learning with verifiable rewards. GAPO adaptively adjusts clipping thresholds based on rol…
-
Self-correction methods fail to improve LLM code generation without verification
A new study on arXiv investigates the effectiveness of self-correction methods for large language models (LLMs) in code generation. Researchers found that while some uncertainty estimation techniques correlate weakly wi…
-
New SR-PPO method improves RL for language models with single rollout
Researchers have developed a new method called Single-Rollout Proximal Policy Optimization (SR-PPO) to address the challenges of estimating token-level advantages in reinforcement learning for language models. This appr…
-
New Local Branch Routing framework enhances language model reasoning
Researchers have developed a new framework called Local Branch Routing (LBR) to improve language model reasoning during test-time scaling. LBR operates at the token level, expanding a local lookahead tree and using a li…
-
New research frames RLVR diversity collapse as overtraining
A new research paper published on arXiv explores the phenomenon of "diversity collapse" in Reinforcement Learning with Verifiable Rewards (RLVR), a technique used to enhance large language models' reasoning. The paper f…