Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
Researchers have identified a critical feature norm threshold, fn*, that largely dictates when neural collapse occurs in deep learning models. This threshold is specific to each model-dataset pair and is largely unaffected by training conditions, though training speed can vary. The study found that crossing this threshold consistently precedes neural collapse, acting as a practical predictor. Factors like network depth, activation functions, weight decay, and width all influence the speed of collapse and the value of fn*. AI
IMPACT Provides a new diagnostic tool for understanding and predicting representational reorganization in deep networks.