A Short and Unified Convergence Analysis of the SAG, SAGA, and IAG Algorithms
Researchers have developed a unified convergence analysis for SAG, SAGA, and IAG algorithms, which are commonly used in large-scale machine learning. This new analysis uses a novel Lyapunov function and concentration tools to establish bounds on delays caused by stochastic sub-sampling. The resulting proof is concise and modular, offering high-probability bounds for SAG and SAGA that can be extended to non-convex objectives. Additionally, this technique yields improved convergence rates for the IAG algorithm. AI
IMPACT Provides a more efficient and unified theoretical understanding for optimization algorithms used in large-scale machine learning.