Researchers have demonstrated that the low-rank subspace assumption used in memory-efficient optimizers like GaLore for training large language models is not as stable as previously believed. Their analysis shows that the projected gradient subspace, recomputed at intervals, is dominated by noise and does not consistently track a slowly drifting object, especially as model scale increases. The study suggests that instead of tracking the subspace, treating each refresh as a coordinate change for optimizers like Adam, particularly with the LDAdam approach, yields better results. AI
IMPACT Challenges the effectiveness of current memory-efficient training techniques, potentially guiding future optimizer development for larger models.
RANK_REASON The cluster contains an academic paper detailing novel research findings on LLM training methodologies.
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
