PulseAugur
EN
LIVE 10:51:01

Adam vs. SGD: No single factor explains performance gap, study finds

By PulseAugur Editorial · [2 sources] ·

A new research paper explores the performance gap between the Adam and SGD optimization algorithms, finding that no single factor consistently explains the difference. The study indicates that the gap arises from complex interactions between data and model architecture, rather than a solitary cause. Researchers observed a crossover batch size where the advantage shifts between Adam and SGD as batch size increases, a phenomenon captured by their theoretical model. AI

IMPACT This research reconciles existing hypotheses on optimization algorithm performance and offers practical insights for training models across various domains.

RANK_REASON The cluster contains a research paper published on arXiv detailing empirical and theoretical findings on AI optimization algorithms.

Read on arXiv cs.LG →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni, Jun Pang, Aurelien Lucchi, Antonio Orvieto ·

    Beyond a Single Explanation of the Adam--SGD Gap

    arXiv:2606.14259v1 Announce Type: new Abstract: Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties. Yet these explanations are often studied in isolatio…

  2. arXiv cs.LG TIER_1 English(EN) · Antonio Orvieto ·

    Beyond a Single Explanation of the Adam--SGD Gap

    Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties. Yet these explanations are often studied in isolation, leaving their relative importance unclear. In…

RELATED ENTITIES

RELATED TOPICS