pass@k
PulseAugur coverage of pass@k — every cluster mentioning pass@k across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
New research highlights limitations in AI evaluation methods
A new paper explores the limitations of pass@k evaluations in machine learning, particularly when extrapolating beyond the number of samples collected. The research demonstrates that fixed-n success counts in conditiona…
-
New DATPO method enhances reasoning coverage in Large Reasoning Models
Researchers have developed DATPO, a new method to improve the reasoning capabilities of Large Reasoning Models trained with Reinforcement Learning with Verifiable Rewards (RLVR). DATPO addresses the limitation of RLVR i…
-
New Method Boosts Natural Language to Logic Translation Accuracy
Researchers have developed a new method called Stratified Consistency Distillation to improve the accuracy of translating natural language into logical formulas. This approach uses a frontier LLM to generate multiple lo…
-
New RL method GAPO boosts Qwen and Llama performance on benchmarks
Researchers have introduced Group Adaptive Clipping Policy Optimization (GAPO), a novel method designed to enhance reinforcement learning with verifiable rewards. GAPO adaptively adjusts clipping thresholds based on rol…
-
New RLVR method boosts LLM reasoning diversity and performance
Researchers have developed a new method to enhance the reasoning capabilities of large language models (LLMs) while maintaining their diversity. The approach, termed Reinforcement Learning with Verifiable Rewards (RLVR)…
-
AI sampling temperature theory explains budget-dependent optimization
Researchers have developed a theoretical framework to explain why the optimal sampling temperature for AI models often increases with a larger sampling budget. This phenomenon, observed across various models like Codex,…
-
New research identifies pass@k inversion in RLVR, proposes mitigation strategy
A new research paper explores the phenomenon of "pass@k inversion" in reinforcement learning with verifiable rewards (RLVR). This occurs when RLVR improves a model's one-sample accuracy but degrades its performance on t…
-
New research tackles AI code generation evaluation and testing
Two new research papers explore advancements in evaluating AI-generated code. The first, TENET, introduces a framework for repository-level code generation using test-driven development, achieving high Pass@1 scores on …
-
New research questions LLM fine-tuning effectiveness for Dart code decompilation
A new research paper explores the effectiveness of fine-tuning large language models for the neural decompilation of Dart Ahead-of-Time (AOT) binaries. The study found that fine-tuning did not significantly improve pass…
-
New SR-PPO method improves RL for language models with single rollout
Researchers have developed a new method called Single-Rollout Proximal Policy Optimization (SR-PPO) to address the challenges of estimating token-level advantages in reinforcement learning for language models. This appr…
-
New research frames RLVR diversity collapse as overtraining
A new research paper published on arXiv explores the phenomenon of "diversity collapse" in Reinforcement Learning with Verifiable Rewards (RLVR), a technique used to enhance large language models' reasoning. The paper f…
-
New RLVR method tackles code generation redundancy
Researchers have developed a new method called Redundancy-Aware RLVR to improve code generation from large language models. This approach addresses the issue of generated code samples being too similar to each other, wh…