New CAST method improves LLM pretraining by adaptively clipping gradients

Researchers have developed CAST (Clipping with Adaptive Scales and Thresholds), a novel method for LLM pretraining that builds upon soft spectral clipping. CAST adaptively adjusts gradient scales and clipping thresholds based on the gradient's history and estimated spectral norm. When applied to a 19.08B parameter Mixture of Experts model trained on 83.89B tokens, CAST demonstrated superior performance over nine other gradient-processing techniques, achieving lower training and validation losses and reducing gradient spikes. This method also enabled the model to reach the best baseline's final smoothed training loss in fewer steps. AI

IMPACT This new pretraining technique could lead to more efficient and stable training of large language models, potentially reducing computational costs and accelerating development.

RANK_REASON The cluster describes a new method for LLM pretraining presented in an arXiv paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• AdamW
• arXiv
• CAST
• Hugging Face
• LLM Pretraining
• Mixture of Experts (MoE)
• Soft spectral clipping (SSC)

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →