LoRA rank allocation fails in RL fine-tuning, study finds

A new study on the Qwen 2.5 1.5B model reveals that adaptive rank allocation techniques, effective in supervised fine-tuning, do not translate to reinforcement learning with Group Relative Policy Optimization (GRPO). Researchers found that proportional rank allocation under GRPO decreased accuracy by 4.5 percentage points compared to uniform allocation. This is attributed to a flatter gradient landscape in GRPO, where all layers retain meaningful gradient signals, and a gradient amplification effect that further widens importance disparities, silencing lower-rank layers. AI

IMPACT Findings suggest current fine-tuning methods for supervised learning may not directly apply to alignment training, potentially requiring new approaches for RL-based fine-tuning.

RANK_REASON Academic paper detailing empirical study of model fine-tuning techniques. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Group Relative Policy Optimization (GRPO)
• GSM8K
• LoRA
• Qwen 2.5 1.5B

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →