A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM$Δ$ Integration into Upcycled MoE
Researchers have developed a new method to efficiently expand Large Language Models (LLMs) to support more languages without extensive retraining. The technique involves converting a dense model into a Mixture-of-Experts (MoE) architecture, with different experts handling different languages. This approach allows for the integration of new language capabilities through post-training parameter deltas, bypassing the need for complex alignment phases and preserving the model's original abilities. AI
IMPACT This method could significantly reduce the cost and complexity of making LLMs multilingual, potentially accelerating global access to advanced AI capabilities.
