New Li$^{2}$SAO method enhances LLM training efficiency

Researchers have introduced Li$^{2}$SAO, a novel asynchronous reinforcement learning optimization method designed to improve the training of large language models (LLMs). This new approach addresses limitations in existing methods like GRPO and SAO, particularly during the early stages of training when value models are not yet well-developed. Li$^{2}$SAO utilizes group-relative actor-critic bootstrapping to train the actor and critic effectively, even with a 'cold' critic, and then transitions to single-rollout GAE for later stages. Experiments demonstrate that Li$^{2}$SAO matches or exceeds GRPO's performance in early training and achieves a higher performance ceiling in later stages, especially on agentic tasks. AI

IMPACT This new method could improve the efficiency and performance of training large language models, particularly for agentic tasks.

RANK_REASON This is a research paper detailing a new method for reinforcement learning optimization in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• GRPO
• large language models
• Li$^{2}$SAO
• reinforcement learning
• Zichen Bai

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →