Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining
Researchers have developed a novel post-hoc method called DG-Hard to address catastrophic forgetting in language models. This technique aims to recover lost capabilities after fine-tuning without requiring retraining, by analyzing the spectral properties of the model's weight updates. DG-Hard applies a singular-value decomposition filtering step to isolate and retain beneficial changes while removing residual noise, demonstrating strong performance across various benchmarks and even restoring safety alignment. AI
IMPACT Offers a potential solution to catastrophic forgetting, enabling more efficient fine-tuning and preservation of model capabilities.