Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
A new research paper analyzes catastrophic forgetting in large language models during continual fine-tuning, comparing twenty leading models. The study categorizes its investigation into behavioral analysis of closed-source models like Claude Fable 5 and GPT 5.5 High, and mechanistic interpretation of open-weight models such as DeepSeek V4-Pro and Llama 4 Maverick. Researchers identified that early-layer attention heads show dispersion while mid-to-deep feed-forward networks experience localized collapse. To address this, they propose Low-Rank Circuit Projection (LRCP), an intervention that successfully mitigates up to 94.2% of ancestral capability loss in open-weight models. AI
IMPACT Proposes a new intervention to mitigate catastrophic forgetting, potentially improving LLM adaptability and performance in continual learning scenarios.