Maximal Update Parametrization
PulseAugur coverage of Maximal Update Parametrization — every cluster mentioning Maximal Update Parametrization across labs, papers, and developer communities, ranked by signal.
-
New framework predicts hyperparameter transfer laws for neural networks
Researchers have developed a new framework called Hyperparameter Transfer Laws to better understand and predict how hyperparameters should be adjusted when scaling neural network architectures. This framework introduces…
-
Gated Delta Networks scaling rules improve LLM training stability
Researchers have developed new scaling rules for Gated Delta Networks, a type of neural network architecture. These rules, derived through a method called coordinate-size estimation propagation, allow for stable learnin…
-
Probabilistic Transformer scales to 0.4B parameters, outperforming standard models
Researchers have developed a method to scale Probabilistic Transformers (PTs) by transferring hyperparameters from smaller models to larger ones using Maximal Update Parametrization (muP). This technique addresses PT's …