McCloskey and Cohen

Catastrophic forgetting in LLMs: How fine-tuning erodes capabilities

Fine-tuning large language models can lead to catastrophic forgetting, where a model loses previously acquired capabilities when optimized for a new objective. This phenomenon, rooted in gradient descent, causes the mod…