Depth in neural networks induces implicit low-rank bias, study finds

Researchers have explored the implicit bias of depth in neural networks, specifically within the deep unconstrained feature model (UFM). Their analysis, focusing on gradient descent and depth without explicit regularization, reveals that depth inherently promotes a low-rank bias. This bias encourages solutions that deviate from standard neural collapse, aligning instead with max-margin solutions previously observed in width-bottlenecked networks. The study also identifies how spectral initialization influences singular values and characterizes the shrinking basin of attraction for neural collapse as depth increases. AI

IMPACT Provides theoretical insights into the behavior of deep neural networks, potentially influencing future model architectures and training methodologies.

RANK_REASON Academic paper published on arXiv detailing theoretical findings about neural network behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Deep Unconstrained Feature Model
• Neural Collapse

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →