New research identifies pass@k inversion in RLVR, proposes mitigation strategy

A new research paper explores the phenomenon of "pass@k inversion" in reinforcement learning with verifiable rewards (RLVR). This occurs when RLVR improves a model's one-sample accuracy but degrades its performance on tasks requiring multiple attempts (higher k). The paper argues this is an absence-of-evidence failure, where rare correct trajectories are lost during RLVR training. A proposed method, Per-Problem Base Anchoring (PBA), aims to mitigate this by sharpening prompts and anchoring risky ones to the base model's distribution, showing improved results on benchmarks like Omni-MATH-Test. AI

IMPACT This research could lead to more robust AI models by addressing a specific failure mode in verifiable reward systems, potentially improving performance in complex reasoning tasks.

RANK_REASON Academic paper detailing a novel research finding and proposed method. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• European Conference on Computer Vision
• Grpo
• MATH500
• Omni-MATH-Test
• pass@k
• RLVR

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →