Benjamin Thérien

MuLoCo framework enhances LLM training with Muon optimizer

Researchers have introduced MuLoCo, a new framework designed to optimize the training of large language models (LLMs) within the DiLoCo system. MuLoCo addresses performance degradation observed in DiLoCo as the number o…