PulseAugur
EN
LIVE 15:03:18

New research probes SAM optimizer's stability and adaptive learning

By PulseAugur Editorial · [3 sources] ·

Two new research papers delve into the complexities of Sharpness-Aware Minimization (SAM), a popular deep learning training technique. The first paper analyzes SAM's convergence instability near saddle points, theoretically proving that it can become an attractor and that momentum and batch-size may be crucial for mitigating this issue. The second paper introduces adaptive Polyak-type step size schedulers specifically for SAM, aiming to reduce the need for extensive learning rate tuning while maintaining or improving performance. AI

IMPACT These papers offer theoretical insights and practical improvements for SAM, potentially leading to more stable and efficient deep learning model training.

RANK_REASON Two academic papers published on arXiv discussing theoretical aspects and improvements of a machine learning optimization technique.

Read on arXiv cs.AI →

AI-generated summary · Google Gemini · from 3 sources. How we write summaries →

COVERAGE [3]

  1. arXiv cs.AI TIER_1 English(EN) · Hoki Kim, Jinseong Park, Yujin Choi, Jaewook Lee ·

    Stability Analysis of Sharpness-Aware Minimization

    arXiv:2301.06308v2 Announce Type: replace-cross Abstract: Sharpness-aware minimization (SAM) is a training method that seeks to find flat minima in deep learning, resulting in state-of-the-art performance across various domains. Instead of minimizing the loss of the current weigh…

  2. arXiv stat.ML TIER_1 English(EN) · Dimitris Oikonomou, Nicolas Loizou ·

    Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler

    arXiv:2606.01827v1 Announce Type: cross Abstract: Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models. By explicitly minimizing the sharpness of the loss landscape, SAM often improves generaliza…

  3. arXiv stat.ML TIER_1 English(EN) · Nicolas Loizou ·

    Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler

    Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models. By explicitly minimizing the sharpness of the loss landscape, SAM often improves generalization while delivering strong empirical performance…

RELATED ENTITIES

RELATED TOPICS