PulseAugur
EN
LIVE 16:39:42

Research paper details how learning-rate cooldown effectiveness depends on noise and optimizer normalization

By PulseAugur Editorial · [2 sources] ·

A new research paper explores the effectiveness of the learning-rate cooldown phase in large-model pretraining, a common component of Warmup-Stable-Decay (WSD) schedules. The study reveals that the benefit of this cooldown is contingent on the interplay between the structure of gradient noise and whether the optimizer employs normalization. Without normalization, stochastic gradient descent (SGD) naturally contracts and benefits little from cooldown. However, normalized methods and SignSGD can get stuck in a noise floor, where cooldown might offer improvements depending on the specific noise characteristics. AI

IMPACT Provides theoretical insights into optimizing large model training, potentially leading to more efficient pretraining methods.

RANK_REASON Research paper published on arXiv detailing theoretical findings about optimization schedules.

Read on arXiv cs.LG →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

Research paper details how learning-rate cooldown effectiveness depends on noise and optimizer normalization

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Subham Singh, Ashutosh Mishra, Subha Raut ·

    Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

    arXiv:2607.12360v1 Announce Type: new Abstract: The cooldown phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining, lowers the final training loss in some settings and does nothing in others. We give a provable account of which case…

  2. arXiv cs.LG TIER_1 English(EN) · Subha Raut ·

    Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

    The cooldown phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining, lowers the final training loss in some settings and does nothing in others. We give a provable account of which case obtains, and it turns on two properties togethe…

RELATED ENTITIES

RELATED TOPICS