Linearized ReLU network

Spectral collapse hinders deep learning plasticity, researchers find

Researchers have identified spectral collapse as a key reason why deep neural networks lose plasticity when learning new tasks. This phenomenon occurs when the Hessian matrix loses effective curvature, rendering gradien…