Researchers have explored the implicit bias of depth in neural networks, specifically within the deep unconstrained feature model (UFM). Their analysis, focusing on gradient descent and depth without explicit regularization, reveals that depth inherently promotes a low-rank bias. This bias encourages solutions that deviate from standard neural collapse, aligning instead with max-margin solutions previously observed in width-bottlenecked networks. The study also identifies how spectral initialization influences singular values and characterizes the shrinking basin of attraction for neural collapse as depth increases. AI
Summary written by gemini-2.5-flash-lite from 1 sources. How we write summaries →
IMPACT Provides theoretical insights into the behavior of deep neural networks, potentially influencing future model architectures and training methodologies.
RANK_REASON Academic paper published on arXiv detailing theoretical findings about neural network behavior. [lever_c_demoted from research: ic=1 ai=1.0]