Schedule-Free AdamW leads AlgoPerf V1 benchmark for deep learning training algorithms

A new benchmark called AlgoPerf V1 has evaluated deep learning training algorithms, focusing on wall-clock time-to-target across various workloads. Schedule-Free AdamW emerged as the top performer among 13 submissions in PyTorch and JAX, utilizing a single configuration across all tasks. The study also highlighted that while some methods offer good sample efficiency, they require significant computational optimization for competitive runtime. Furthermore, analyses indicated that learning rate and momentum are critical hyperparameters, underscoring the need for more robust and generalizable training methods. AI

IMPACT Highlights the importance of efficient and robust training algorithms for deep learning, potentially guiding future research and development.

RANK_REASON Research paper presenting new benchmark results and analysis of training algorithms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• AlgoPerf
• arXiv
• Jax
• PyTorch
• Schedule-Free AdamW

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →