New method enhances reinforcement learning for LLMs by recovering learning signals

Researchers have developed a new method called RSTG (Recovering Learning Signals via Adaptive Teacher Guidance) to improve reinforcement learning for large language models. This technique addresses the issue of sparse reward signals in methods like GRPO by selectively applying distillation from a teacher model. RSTG focuses distillation on negative zero-variance prompts, weights samples by teacher confidence, and targets specific tokens with high student entropy or large teacher-student divergence. Additionally, it incorporates supervised fine-tuning on teacher-generated correct trajectories to inject positive gradient signals where RL fails. Experiments show RSTG significantly outperforms standard GRPO+OPD on math and code tasks. AI

IMPACT This research could lead to more efficient and effective training of large language models, improving their performance on complex tasks like math and coding.

RANK_REASON The cluster contains a research paper detailing a new method for improving LLM training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• GRPO
• RLVR
• RSTG
• supervised fine-tuning

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →