A new arXiv paper investigates the impact of switching optimizers during large language model (LLM) training. Researchers found that performance is largely determined by the independent effects of the pretraining and post-training optimizers, rather than a significant penalty for mismatch. Specifically, increasing the proportion of updates with the Muon optimizer during supervised fine-tuning (SFT) generally improved downstream performance, even when models were pretrained with AdamW. While matching the pretraining optimizer can reduce forgetting during SFT, it did not lead to better final benchmark results. AI
IMPACT Suggests flexibility in choosing optimizers for post-training stages of LLMs without significant performance degradation.
RANK_REASON Academic paper on LLM training methodology. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
