LDAdam

New research questions stability of low-rank training for LLMs

Researchers have demonstrated that the low-rank subspace assumption used in memory-efficient optimizers like GaLore for training large language models is not as stable as previously believed. Their analysis shows that t…