The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
Researchers have explored the implicit bias of depth in neural networks, specifically within the deep unconstrained feature model (UFM). Their analysis, focusing on gradient descent and depth without explicit regularization, reveals that depth inherently promotes a low-rank bias. This bias encourages solutions that deviate from standard neural collapse, aligning instead with max-margin solutions previously observed in width-bottlenecked networks. The study also identifies how spectral initialization influences singular values and characterizes the shrinking basin of attraction for neural collapse as depth increases. AI
IMPACT Provides theoretical insights into the behavior of deep neural networks, potentially influencing future model architectures and training methodologies.