Researchers have developed a theoretical framework and simulation algorithm to model the training dynamics of Deep Neural Networks (DNNs), specifically addressing the phenomenon of intermittent loss spikes observed during the pre-training of Large Language Models (LLMs). Their work demonstrates that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate these spikes. In a case study involving an Adam hyperparameter configuration known to induce spikes in Transformer LMs, their simulation successfully predicted spiky behavior from a given checkpoint, and they derived a $eta_2$ schedule that empirically improved performance. AI
IMPACT Provides a theoretical framework to understand and potentially mitigate loss spikes during LLM pre-training.
RANK_REASON The cluster contains a research paper detailing a new theoretical framework and simulation algorithm for modeling DNN training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]
- Adam
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- Deep Neural Networks
- Gotit.pub
- Hugging Face
- IArxiv
- large-language models
- ScienceCast
- transformer LMs
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
