DASH optimizer speeds up Shampoo by up to 5.6x with GPU and root-finding innovations

Researchers have developed DASH, a significantly faster implementation of the Shampoo optimizer for machine learning. DASH utilizes batched block preconditioning to improve GPU utilization and introduces novel methods like Newton-DB and Chebyshev polynomial approximations for computing inverse matrix roots. This optimization results in up to a 5.6x speedup in optimizer steps compared to existing Distributed Shampoo implementations, while also achieving lower validation perplexity per iteration. AI

IMPACT Accelerates training of large machine learning models by improving optimizer efficiency.

RANK_REASON Academic paper detailing a new method for optimizing machine learning algorithms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• GPU
• Ionut-Vlad Modoranu
• MLCommons AlgoPerf
• Newton-DB
• Shampoo

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →