Researchers have developed a new objective function called Minimum Prefix Ratio (MinPRO) to stabilize off-policy reinforcement learning for large language models (LLMs). Existing methods using token-level importance ratios struggle with training stability when there's a significant lag between the sampling and target policies. MinPRO addresses this by using a surrogate based on the minimum token-level ratio within a prefix, demonstrating improved stability and performance on mathematical reasoning benchmarks for both dense and mixture-of-experts LLMs. AI
IMPACT This research could lead to more stable and efficient training of LLMs for complex reasoning tasks.
RANK_REASON The cluster contains an academic paper detailing a new method for LLM optimization. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- CatalyzeX
- DagsHub
- Hugging Face
- large language models (LLMs)
- Minimum Prefix Ratio (MinPRO)
- reinforcement learning
- Shiye Lei
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
