New research shows Transformer training trajectories diverge despite identical functions

A new research paper explores the phenomenon where identical Transformer models can exhibit different training trajectories due to optimizer behavior. The study demonstrates that standard optimizers like AdamW do not adequately account for scale imbalances that arise from parameter transformations, even when the model's function remains unchanged. To address this, the researchers propose methods for balancing matrix pairs and transforming their moment estimates together, ensuring equivalent checkpoints lead to consistent training paths. They also introduce Symmetry-Aware Gradient Geometry Stabilization (SAGGS) and its variant Soft SAGGS, which equalize Frobenius norms of symmetry pairs to mitigate scale imbalance, showing significant perplexity reductions on WikiText-103. AI

IMPACT Identifies a fundamental issue in Transformer training stability, potentially leading to more robust and reproducible model development.

RANK_REASON Academic paper detailing novel research findings on model training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.NE (Neural & Evolutionary) →

• AdamW
• Hugging Face
• Soft SAGGS
• Symmetry-Aware Gradient Geometry Stabilization
• transformer
• WikiText-103

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →