Maximal Update Parameterization
PulseAugur coverage of Maximal Update Parameterization — every cluster mentioning Maximal Update Parameterization across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
ShellElastic offers scalable elastic pretraining for foundation models
Researchers have developed ShellElastic, a novel approach to training foundation models that efficiently supports diverse inference budgets. This method trains nested sub-networks within a single supernet, overcoming ch…
-
Compute-efficient hyperparameter transfer for large MoE models
Researchers have developed a compute-efficient framework for determining optimal learning rates in large Mixture-of-Experts (MoE) models. This two-step process involves transferring optimal learning rates across models …
-
New framework efficiently predicts optimal learning rates for large MoE models
Researchers have developed a novel two-step framework to efficiently determine optimal learning rates for large Mixture-of-Experts (MoE) models. This method leverages hyperparameter transfer across different model width…