New ORBIT method prevents LLM catastrophic forgetting during fine-tuning

Researchers have introduced ORBIT, a new method designed to prevent large language models from losing their foundational language capabilities during task-specific fine-tuning. This issue, known as catastrophic forgetting, is particularly prevalent in Generative Retrieval tasks and is linked to the divergence of model parameters. ORBIT addresses this by monitoring the distance between fine-tuned and original model weights, employing a weight averaging strategy to limit parameter drift when a set threshold is exceeded. Experiments demonstrate that ORBIT effectively preserves text and retrieval performance, outperforming existing continual learning and regularization techniques. AI

IMPACT Preserves general language abilities during task-specific LLM fine-tuning, potentially improving model versatility.

RANK_REASON Publication of an academic paper introducing a novel method for LLM fine-tuning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• LLM
• ORBIT

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →