Sham Kakade

Harvard researchers unveil simple quadratic model predicting LLM pre-training dynamics

Researchers at Harvard University have developed a simple quadratic model that accurately predicts the optimization dynamics of large language models during pre-training. By applying Taylor's theorem to real neural netw…