Brief

Researchers have developed Ringmaster LMO, a novel asynchronous method for training neural networks that addresses inefficiencies in distributed systems. This approach builds upon the delay-thresholding concept to manage gradient staleness, aiming to improve training speed in heterogeneous environments. The method is designed for unconstrained stochastic non-convex optimization and has demonstrated superior performance compared to existing synchronous and asynchronous baselines in experiments involving quadratic problems and language model pretraining. AI

Researchers have introduced a new principle for designing optimizers in deep learning that aligns with the inherent symmetries of neural network architectures. Unlike current optimizers like Adam, which operate on parameters in a coordinate-wise manner, the proposed symmetry-compatible optimizers are designed to be equivariant to the specific symmetry groups of different weight blocks. This approach has been applied to various components such as embeddings, LM heads, MLPs, and MoE routers, yielding novel update rules. Experiments on language models demonstrate that these new optimizers consistently improve validation loss and training stability compared to standard AdamW. AI

Researchers have developed new methods for hyperparameter transfer, enabling more efficient scaling of large neural networks. One paper introduces a parameterization justified by dynamical mean-field theory, allowing reliable hyperparameter transfer across models ranging from 51 million to over 2 billion parameters. Another study quantifies hyperparameter transfer and highlights the critical role of the embedding layer's learning rate, suggesting that maximizing it can significantly improve training stability and performance, particularly when using the AdamW optimizer. AI

Researchers have developed SF-NorMuon, a new schedule-free spectral optimizer that matches or surpasses the performance of traditional AdamW optimizers. This advancement addresses a key limitation in current anytime training methods, where schedule-free approaches often underperform. SF-NorMuon's ability to achieve high-quality training checkpoints at any point without pre-defined horizons makes it a more practical tool for open-ended continual learning. AI

A new research paper demonstrates that the choice of optimizer significantly impacts a Transformer model's capacity and scaling laws, even when the architecture remains identical. The study found that the Muon optimizer achieved linear scaling in representation capacity, a 2.3x improvement over AdamW's weaker scaling, particularly in challenging rare-token regimes. This suggests that optimizers should be considered a primary factor in model scaling, alongside architecture and data, and highlights the potential for co-designing optimizers and architectures for better performance. AI

Researchers have developed several new optimization techniques to improve deep learning model training. AMUSE combines the rapid adaptation of Muon with the stability of Schedule-Free averaging, eliminating the need for learning rate schedules and improving performance across vision and language tasks. Another approach, MiMuon, enhances the generalization capabilities of Muon by blending it with SGD, offering a lower generalization error. Additionally, a new optimizer called Pion addresses Muon's limitations in vision-language-action and reinforcement learning by employing a spectral high-pass filtering mechanism. AI

Several recent research papers explore advanced optimization techniques for machine learning. One paper introduces a derivative-free consensus-based method for nonconvex bi-level optimization, demonstrating convergence guarantees for its mean-field and finite-particle approximations. Another study presents Curvature-Tuned Accelerated Gradient Descent (CT-AGD), which reduces training epochs by an average of 33% for deep learning tasks by capturing local curvature. Additionally, research investigates stochastic approximation algorithms under heavy-tailed noise, analyzing concentration bounds and the impact of noise on error tails. Other papers delve into stochastic gradient variational inference, global convergence of stochastic conic particle gradient descent, and the suboptimality of momentum SGD in nonstationary environments. AI