Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates
Researchers have developed a new method called FINCH to address catastrophic forgetting during the fine-tuning of large language models. FINCH employs a loss-adaptive learning rate schedule that decreases the learning rate for high-loss batches and increases it as the model converges. This approach effectively reduces forgetting by an average of 93% across various benchmarks while maintaining task performance. FINCH also shows improvements in preserving model calibration and confidence. AI
IMPACT FINCH significantly reduces catastrophic forgetting in LLMs, potentially enabling more effective and stable fine-tuning for specialized tasks.