FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Researchers have introduced FoMoE, a novel system designed to overcome the limitations of training large language models (LLMs) across geographically distributed data centers. Unlike previous methods that required full model replicas at each site, FoMoE partitions expert layers across workers, significantly reducing communication costs and memory overhead. This approach enables more efficient scaling of LLMs, achieving empirical throughput speedups and projecting substantial benefits for models up to 100 billion parameters. AI
IMPACT Enables more efficient and scalable training of large language models across distributed, weakly connected data centers.