Researchers have identified a precise mathematical law governing the instability of scale-invariant optimization in neural networks. This law reveals a feedback loop between learning-rate schedules and weight decay, mediated by the parameter norm, which dictates the effective step size of optimizers. The study proposes a single scalar quantity that precisely controls training dynamics and identifies a sharp boundary separating stable and unstable learning rate regimes. This framework offers a new lens for understanding optimizer behavior and designing training schedules, with findings validated across various neural network architectures and datasets, including MLP, CNN, and GPT2. AI
IMPACT Provides a precise, actionable framework for optimizing deep learning training dynamics and schedule design.
RANK_REASON The cluster contains an academic paper detailing new theoretical findings in neural network optimization.
- Cifar
- GPT2
- MNIST database
- normalized-optimization-dynamics
- OpenWebText
- Optimizer
- wikitext
- CNN
- Hugging Face
- learning-rate schedules
- multilayer perceptron
- normalized regression model
- Parameter Norm
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
