New Muon^p Optimizer Enhances Fine-Tuning of Large Models

Researchers have introduced Muon$^p$, an optimization technique that refines the existing Muon optimizer by using fractional spectral-power updates. This method interpolates between full spectral flattening and standard gradient descent, aiming to preserve valuable singular-value information for better adaptation. Muon$^p$ is particularly effective for fine-tuning large-scale models, showing improvements in validation perplexity and downstream task performance, while maintaining a similar computational complexity to Muon. AI

RANK_REASON The cluster contains a research paper detailing a new optimization technique for machine learning models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Muon
• Muon$^p$

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →