Research links learning rates to catastrophic overtraining in LLMs

A new research paper explores how learning rates impact catastrophic overtraining during the supervised fine-tuning (SFT) of large language models (LLMs). The study, published on arXiv, suggests that different learning rates can lead to qualitatively different models even when trained to the same SFT loss. Specifically, the research indicates that learning rate decay can increase the sharpness of a pretrained model, which in turn exacerbates forgetting and overtraining during SFT. AI

IMPACT Provides insights into LLM training dynamics, potentially guiding fine-tuning strategies to mitigate performance degradation.

RANK_REASON Research paper published on arXiv detailing findings on LLM training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• catastrophic overtraining
• learning rate
• learning rate decay
• mathematical optimization
• Springer et al.
• supervised fine-tuning

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →