New Local Support Learning framework combats catastrophic forgetting in LLMs

Researchers have developed Local Support Learning (LSL), a novel framework designed to combat catastrophic forgetting in large pre-trained models. LSL addresses forgetting by treating it as a geometric problem in the weight matrix input space, proposing a retention objective that improves upon standard gradient-based optimizers. The framework integrates a weight adapter with a gating function, which is trained to activate only for inputs from its specific training distribution, thereby localizing updates. This approach has demonstrated success in resolving forgetting in large language models up to 7 billion parameters, preserving both pre-trained and fine-tuned capabilities across multiple training phases with efficient memory and compute usage. AI

IMPACT This new framework could enable more efficient and continuous learning for large language models, reducing the need for extensive retraining.

RANK_REASON Research paper detailing a new method for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gaussian Mixture Model
• Gotit.pub
• Hugging Face
• IArxiv
• Local Support Learning
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →