Adafactor
PulseAugur coverage of Adafactor — every cluster mentioning Adafactor across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New Adaptive Log-Space Quantization Boosts LLM Optimizer Memory Efficiency
Researchers have developed a new quantization method called Adaptive Log-Space (AL) to improve the memory efficiency of optimizers used in training large language models. This method adapts the quantization range per bl…
-
New SkewAdam optimizer slashes MoE training memory by 97%
Researchers have developed SkewAdam, a novel optimizer designed to significantly reduce memory usage during the training of Mixture-of-Experts (MoE) models. By allocating different levels of precision for optimizer stat…
-
New Rose optimizer offers low VRAM, fast convergence, and great results
A new PyTorch optimizer named Rose has been released under the Apache 2.0 license. Developed by Matthew K., Rose is designed to be stateless, offering significantly lower VRAM usage compared to optimizers like AdamW, wi…