LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging
Researchers have introduced LOSCAR-SGD, a novel method for distributed machine learning that addresses communication bottlenecks. This approach combines local training, sparse model updates, and communication-computation overlap to accelerate training, particularly in federated learning scenarios. The method includes a delay-corrected merge rule to effectively integrate synchronized information while optimizing during communication periods. Theoretical convergence guarantees are provided for smooth non-convex objectives, and experimental results demonstrate reduced training times and improved performance over naive methods. AI
IMPACT Optimizes distributed training efficiency, potentially accelerating large-scale AI model development.