New methods tackle memory peaks for long-context MoE training

Researchers have developed four novel techniques to address memory limitations in training Mixture-of-Experts (MoE) models with long contexts. These methods, PipelinedLLEP, Ring-DTP, Selective Checkpoint Offload (SCO), and OffloadStreamAdamW, target specific memory bottlenecks including expert dispatch, vocabulary projections, gradient checkpointing, and optimizer states. By altering computation and data movement order rather than the computations themselves, these techniques maintain exact gradients and loss. When combined, they enable MoE models with up to 667 billion parameters to train at a 1 million token context length, significantly extending reach and improving throughput compared to standard baselines. AI

IMPACT These techniques could enable training of much larger and more capable MoE models with extended context windows.

RANK_REASON The item is a research paper detailing novel techniques for training large AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• AdamW
• FSDP2
• mixture of experts
• OffloadStreamAdamW
• PipelinedLLEP
• Ring-DTP
• Selective checkpoint offload

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →