Quantization Error Drives Language Model Training Instability

Researchers have identified a significant cause of instability during large language model pretraining: loss spikes. These spikes, often emerging late in training and termed "time-bomb spikes," are linked to the growth and divergence of output logits in the model's final language modeling head. The study found that quantization error in this head precedes these spikes, with reduced numerical precision accelerating them and increased precision improving stability. A proposed mitigation involves replacing AdamW with SGD with momentum for the LM head, which suppresses logit growth and enhances stability over extended training periods. Existing mitigation techniques also appear to reduce quantization error, suggesting a common numerical mechanism for their effectiveness. AI

IMPACT Identifies a key numerical instability in LLM pretraining, potentially improving training efficiency and stability.

RANK_REASON Academic paper detailing a specific technical finding and proposed mitigation for LLM training instability. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• AdamW
• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• Litmaps
• ScienceCast
• scite Smart Citations
• SGD

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →