Brief

Researchers have developed a new method to stabilize the training of large Transformer models, which are often prone to instability and divergence. The approach, called "architecture warm-up," involves progressively increasing the network depth to manage the preconditioned Hessian, a measure of curvature that correlates with training instabilities. This technique, supported by a fast online estimator for Hessian eigenvalues, has been shown to reduce instabilities without hindering convergence. AI

Two new research papers introduce novel optimization techniques for deep learning models. The first paper, "Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization," proposes Hyperball, an optimizer wrapper that maintains performance gains with increasing model size by fixing weight matrix norms. The second paper, "OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality," presents OptEMA, an adaptive EMA optimizer that achieves near-optimal rates in zero-noise scenarios without manual hyperparameter tuning. A third paper, "Gefen: Optimized Stochastic Optimizer," introduces Gefen, a memory-efficient optimizer that reduces AdamW's memory footprint by approximately 8x while maintaining performance, enabling larger batch sizes and potentially larger models. AI

A new paper analyzes how overparameterization in neural networks aids optimization by introducing additional symmetries. These symmetries act as a form of preconditioning on the Hessian, leading to better-conditioned minima. Furthermore, overparameterization increases the likelihood of finding global minima near typical initializations, making them more accessible. Experiments with teacher-student networks confirmed these theoretical predictions, showing improved convergence and condition numbers with increased network width. AI