LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Researchers have introduced LionMuon, a novel optimization algorithm designed for efficient training of large-scale models. This method alternates between the low-cost updates of Lion and the stronger, albeit more expensive, spectral updates of Muon. By sharing a single momentum buffer, LionMuon significantly reduces the average iteration cost while maintaining effectiveness. Experiments show LionMuon outperforms existing optimizers like Muon, Lion, Signum, and AdamW across various model sizes and datasets, achieving lower validation loss with less compute. AI
IMPACT Introduces a new optimization technique that could significantly reduce the computational cost of training large AI models.