OffloadStreamAdamW

New methods tackle memory peaks for long-context MoE training

Researchers have developed four novel techniques to address memory limitations in training Mixture-of-Experts (MoE) models with long contexts. These methods, PipelinedLLEP, Ring-DTP, Selective Checkpoint Offload (SCO), …