FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
Researchers have introduced FOAM, a new adaptive algorithm designed to improve the efficiency of the Shampoo optimization method. Shampoo is known for its strong performance on large-scale benchmarks but suffers from high computational costs due to matrix inversion. FOAM addresses this by theoretically analyzing the trade-offs between computational efficiency and optimization fidelity when using stale preconditioner updates. The algorithm dynamically adjusts damping factors and eigendecomposition frequencies to stabilize training and reduce staleness-oriented errors. AI
IMPACT Improves efficiency of large-scale optimization methods, potentially speeding up AI model training.