Lion
PulseAugur coverage of Lion — every cluster mentioning Lion across labs, papers, and developer communities, ranked by signal.
5 day(s) with sentiment data
-
New research unifies Lion and Muon optimizers under Stochastic Frank-Wolfe
Researchers have unified several popular deep learning optimizers, including Lion and Muon, under the framework of Stochastic Frank-Wolfe. This work establishes convergence guarantees for these methods in non-convex opt…
-
New PsiLogic optimizer improves Adam/AdamW performance on benchmarks
Researchers have introduced PsiLogic, a novel optimizer designed to improve upon existing adaptive methods like Adam and AdamW. PsiLogic dynamically adjusts its update rule based on the stability of training, employing …
-
New SkewAdam optimizer slashes MoE training memory by 97%
Researchers have developed SkewAdam, a novel optimizer designed to significantly reduce memory usage during the training of Mixture-of-Experts (MoE) models. By allocating different levels of precision for optimizer stat…
-
LionVote enhances Lion optimizer with per-layer learning rate adaptation
Researchers have developed LionVote, a novel per-layer learning rate adaptation mechanism designed to optimize the Lion optimizer. Diagnostics revealed that Lion's default learning rate is too high for certain parameter…
-
Saber-toothed cat Smilodon had surprisingly weak bite, study finds
New biomechanical modeling of the saber-toothed cat, Smilodon fatalis, reveals its bite force was significantly weaker than previously assumed, comparable to a large house cat rather than a lion. Researchers found that …
-
Optimizers Amplify LLM Misalignment, New Research Finds
A new research paper titled "Evil Spectra" explores emergent misalignment in large language models, finding that the choice of optimizer significantly impacts the rate of misalignment. The study, which tested various Qw…
-
Open problem: AdamW optimizer's effectiveness under heavy-tailed noise in LLMs
A recent paper poses an open problem regarding the effectiveness of the AdamW optimizer in training large language models (LLMs) under heavy-tailed noise conditions. While AdamW is widely used, its theoretical understan…
-
LionMuon optimizer cuts training cost for large models
Researchers have introduced LionMuon, a novel optimization algorithm designed for efficient training of large-scale models. This method alternates between the low-cost updates of Lion and the stronger, albeit more expen…
-
Muown optimizer improves LLM training by controlling row-norm drift
Researchers have developed Muown, a novel optimization method designed to improve the training of large language models. Muown addresses issues with the Muon optimizer, specifically the upward drift of spectral norms in…
-
New LMO-IGT method accelerates optimization with implicit gradient transport
Researchers have introduced LMO-IGT, a novel class of stochastic optimization methods designed to accelerate convergence in machine learning. This approach leverages implicit gradient transport (IGT) to achieve faster r…
-
New Rose optimizer offers low VRAM, fast convergence, and great results
A new PyTorch optimizer named Rose has been released under the Apache 2.0 license. Developed by Matthew K., Rose is designed to be stateless, offering significantly lower VRAM usage compared to optimizers like AdamW, wi…