SILAGE: Memory-Efficient, Full-Gradient-Free Nonconvex Optimization for Nested Finite Sums
Researchers have introduced SILAGE, a novel algorithm designed for memory-efficient, gradient-free nonconvex optimization in machine learning. This method addresses the challenges of empirical risk minimization on large datasets by exploiting a nested double finite-sum structure. Unlike previous approaches that require expensive global gradient refreshes or impractically large memory footprints, SILAGE uses only O(n) memory and avoids periodic global refreshes by evaluating at most one local group gradient per iteration. The algorithm's convergence analysis adapts to data geometry through nested functional similarities, improving upon existing state-of-the-art bounds. AI
IMPACT This new optimization technique could enable more efficient training of large machine learning models on massive datasets.