Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning
Researchers have identified spectral collapse as a key reason why deep neural networks lose plasticity when learning new tasks. This phenomenon occurs when the Hessian matrix loses effective curvature, rendering gradient descent inefficient. The study proposes two regularization techniques—maintaining high effective feature rank and applying L2 penalties—to combat spectral collapse and preserve plasticity in continual learning scenarios. AI
IMPACT Identifies a core mechanism limiting continual learning, suggesting new regularization methods to improve model adaptability.