Harvard researchers unveil simple quadratic model predicting LLM pre-training dynamics

Researchers at Harvard University have developed a simple quadratic model that accurately predicts the optimization dynamics of large language models during pre-training. By applying Taylor's theorem to real neural network checkpoints, they found that higher-order terms in the expansion contribute minimally within the critical pre-training window, suggesting the underlying dynamics are inherently quadratic. This simplified model offers precise insights into core pre-training challenges such as determining optimal stopping times, batch sizes, and learning rates, effectively revealing the essence of the pre-training process. AI

IMPACT This research could lead to more efficient and predictable LLM pre-training by providing a simpler analytical framework for optimization.

RANK_REASON The cluster describes a new research finding and theoretical model presented at a conference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on 雷峰网 (Leiphone) →

• Harvard University
• International Conference on Machine Learning
• Kempner Institute for the Study of Natural and Artificial Intelligence
• linear regression
• LLM
• SGD
• Sham Kakade
• Taylor's theorem

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →