Research reveals LoRA dynamics in SFT vs RL, proposes tailored configurations

A new research paper published on arXiv details the spectral and subspace dynamics of Low-Rank Adaptation (LoRA) during supervised fine-tuning (SFT) and reinforcement learning (RL). The study found that RL concentrates LoRA's spectrum and rotates singular subspaces faster than SFT, with group-mean baseline subtraction identified as the key differentiating factor. Based on these dynamics, the researchers propose tailored LoRA configurations for each stage, showing improved performance on mathematical reasoning tasks across multiple models. AI

IMPACT Provides insights into optimizing LoRA for different training stages, potentially improving model performance in SFT and RL applications.

RANK_REASON Academic paper detailing novel findings on model training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• arXiv
• group-mean baseline subtraction
• Hugging Face
• LoRA
• quadratic multi-index model
• reinforcement learning
• supervised fine-tuning

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →