Catastrophic forgetting in LLMs: How fine-tuning erodes capabilities

Fine-tuning large language models can lead to catastrophic forgetting, where a model loses previously acquired capabilities when optimized for a new objective. This phenomenon, rooted in gradient descent, causes the model's output distribution to collapse towards the fine-tuning data's characteristics. Researchers have identified this issue in models like InstructGPT and explored mitigations such as mixing pretraining gradients or penalizing changes to important parameters, though quantifying the exact extent of forgetting remains challenging and requires task-specific evaluation. AI

IMPACT Fine-tuning LLMs requires careful evaluation to prevent loss of general capabilities, impacting model deployment and reliability.

RANK_REASON The item discusses a research paper and empirical study on catastrophic forgetting in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• arXiv 2308.08747
• Fisher information
• InstructGPT
• McCloskey and Cohen
• Ouyang et al.
• PPO-ptx

• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →