Spes

New SPES framework enables memory-efficient decentralized LLM pretraining on fewer GPUs

Researchers have developed a novel decentralized framework called SPES for pretraining large language models, specifically Mixture-of-Experts (MoE) architectures. This method significantly reduces memory requirements by…