New optimizer decouples weight magnitude and direction for improved neural network training

Researchers have introduced a novel optimization technique called Magnitude--Direction (MD) Decoupling, which separates the magnitude and direction of weight vectors in neural networks. This method allows for independent learning rates for these two components, potentially leading to more stable and efficient training. MD Decoupling has shown improvements over existing optimizers like Adam and Muon, even on large Mixture-of-Experts models, and removes the need for traditional techniques such as weight decay and warmup. AI

IMPACT Could lead to more stable and efficient training of large neural networks, potentially accelerating research and development.

RANK_REASON Academic paper introducing a novel optimization technique for neural networks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• Alexander Hägele
• Magnitude--Direction (MD) Decoupling
• Muon

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →