New research reveals "silent freeze" in low-precision AI training

Researchers have identified a phenomenon called "silent freeze" that occurs during low-precision training of deep learning models. This freeze happens when weight updates round away to zero, effectively halting learning for specific parameters even when gradients are still non-zero. The study demonstrates that this freeze is predictable and can occur in models like GPT-2, impacting their performance. Stochastic rounding has been shown to mitigate this issue. AI

IMPACT Identifies a critical limitation in low-precision training that could affect the efficiency and scalability of future AI models.

RANK_REASON Academic paper detailing a specific technical finding in AI model training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• AdamW
• arXiv
• bfloat16
• CNN
• Fp8
• generative pre-trained transformer
• GPT-2
• Hugging Face
• MNIST database

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →