New research explores composing continual learning mechanisms for long-horizon memorization in language models

Researchers have introduced a new setting called long-horizon memorization to study how language models can retain information over extended periods and numerous updates. Existing continual learning mechanisms struggle with this challenge, leading to significant forgetting. The study proposes composing complementary mechanisms, such as data, function, and weight anchors, alongside low-rank allocation rules like LoRA+, to preserve prior information and manage successive updates. This combined approach demonstrated a substantial improvement in retention, raising it from 1.2% to 34.9% on average across three distinct datasets. AI

IMPACT Proposes a novel approach to improve long-term memory retention in language models, crucial for applications requiring continuous learning.

RANK_REASON Academic paper introducing a new research setting and proposing methods for continual learning in language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• catastrophic forgetting
• continual learning mechanism
• continual supervised fine-tuning
• Data anchors
• Function anchors
• Language Models
• long-horizon memorization
• LoRA+
• Low-rank allocation rules
• sequential fine-tuning
• Weight anchors

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →