Brief

Researchers have explored the implicit bias of depth in neural networks, specifically within the deep unconstrained feature model (UFM). Their analysis, focusing on gradient descent and depth without explicit regularization, reveals that depth inherently promotes a low-rank bias. This bias encourages solutions that deviate from standard neural collapse, aligning instead with max-margin solutions previously observed in width-bottlenecked networks. The study also identifies how spectral initialization influences singular values and characterizes the shrinking basin of attraction for neural collapse as depth increases. AI

Researchers have introduced new methods, NTCE and NONL, to improve supervised classification by achieving Neural Collapse (NC) more efficiently. These techniques address limitations in existing paradigms like cross-entropy and supervised contrastive learning. By treating supervised learning as prototype learning on a hypersphere, the new losses enable faster convergence to NC and yield significant improvements in transfer learning and robustness, especially under class imbalance. AI