New SPES framework enables memory-efficient decentralized LLM pretraining on fewer GPUs

Researchers have developed a novel decentralized framework called SPES for pretraining large language models, specifically Mixture-of-Experts (MoE) architectures. This method significantly reduces memory requirements by training only a subset of experts on each node and synchronizing knowledge efficiently across distributed GPUs, even over internet connections. SPES has demonstrated its capability by successfully training models up to 9 billion parameters, achieving performance comparable to centrally trained models within similar computational budgets. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Introduces a memory-efficient decentralized training paradigm that could lower the hardware barrier for developing large language models.

RANK_REASON Academic paper detailing a new method for distributed LLM pretraining. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• SPES
• LLM
• MoE
• GPU
• H100
• A100
• Jinrui Zhang

• paper
• infra