Warmup-stable-decay
PulseAugur coverage of Warmup-stable-decay — every cluster mentioning Warmup-stable-decay across labs, papers, and developer communities, ranked by signal.
-
New research details learning-rate schedules for least-squares regression
A new research paper published on arXiv analyzes the asymptotic risk of learning-rate schedules for least-squares regression with discretely distributed features. The study focuses on the last iterate of stochastic grad…
-
Research Explores Momentum's Role in LLM Loss Landscape Optimization
A new research paper explores the 'river-valley' structure of loss landscapes in large language models, proposing that momentum acceleration plays a crucial role in optimizing these complex environments. The study sugge…
-
Optimal Learning Rate Schedules Explored Under Functional Scaling Laws
This paper explores optimal learning rate schedules for machine learning models, particularly within the Functional Scaling Law (FSL) framework. It identifies a critical transition point based on task difficulty and mod…
-
New WSqD learning rate schedule offers horizon-free training for large models
Researchers have introduced WSqD, a novel learning rate schedule designed for large model training that is independent of the training horizon. Unlike existing methods like cosine annealing and Warmup-stable-decay (WSD)…