Maximal Update Parameterization

Compute-efficient hyperparameter transfer for large MoE models

Researchers have developed a compute-efficient framework for determining optimal learning rates in large Mixture-of-Experts (MoE) models. This two-step process involves transferring optimal learning rates across models …