Muon optimizer tackles task interference in continual learning and model merging

Researchers have introduced a new perspective on continual learning and model merging, framing both as instances of "task interference." This interference, quantified by a layer-wise Frobenius inner product, is influenced by the base optimizer. The study identifies the spectral norm of parameter updates as a key factor controllable by the optimizer, with the Muon optimizer demonstrating an ability to regulate this factor. Experiments show that replacing AdamW with Muon leads to significant accuracy improvements on model merging benchmarks and consistent gains in continual learning scenarios. AI

IMPACT Introduces a novel optimizer-centric approach to address fundamental challenges in continual learning and model merging, potentially improving model performance and efficiency.

RANK_REASON Academic paper introducing a new theoretical framework and empirical validation for an optimizer. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• AdamW
• arXiv
• continual learning
• Muon

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →