New HybridSGD method optimizes distributed-memory AI training

Researchers have developed HybridSGD, a novel 2D parallel stochastic gradient descent method designed to optimize performance in distributed-memory systems. This new approach offers a continuous trade-off between existing 1D methods like s-step SGD and Federated SGD with Averaging (FedAvg). Theoretical analysis confirms HybridSGD's advantages in convergence, computation, communication, and memory usage. Empirical evaluations on a Cray EX supercomputing system demonstrated that HybridSGD achieves better convergence than FedAvg and significant speedups over both s-step SGD and FedAvg when applied to binary classification tasks. AI

IMPACT This research could lead to more efficient training of large AI models on distributed computing systems.

RANK_REASON The cluster contains an academic paper detailing a new algorithm for distributed optimization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• C++
• Cray EX
• FedAvg
• Federated SGD with Averaging
• HybridSGD
• LIBSVM
• MPI
• s-step SGD
• Stochastic Gradient Descent

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →