Adam and Muon optimizers show implicit bias in neural networks

Researchers have analyzed the implicit bias of momentum-based optimizers like Adam and Muon when applied to smooth homogeneous neural networks. Their findings suggest that algorithms such as momentum steepest descent, including Muon, MomentumGD, and Signum, act as approximate steepest descent trajectories under specific learning rate schedules. This bias leads these algorithms to favor KKT points of the corresponding margin maximization problem, with Adam specifically maximizing the L-infinity margin. AI

IMPACT Provides theoretical insights into optimizer behavior, potentially guiding future model training strategies.

RANK_REASON This is a research paper published on arXiv detailing theoretical analysis and experimental results of optimizers in neural networks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• Eitan Gronich
• MomentumGD
• Muon
• Signum

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →