New research reveals critical initialization destabilizes higher derivatives in neural networks

Researchers have identified that critical initialization can destabilize higher-order input derivatives in wide neural networks with scalar inputs. While the edge-of-chaos condition preserves first-order perturbations, losses and regularizations relying on higher derivatives are affected. The study derives mean-field recursions that are exact at the variance fixed point, showing that second-derivative variance grows linearly with depth when activation functions have non-zero curvature. For residual networks, it's proven that all finite derivative orders maintain uniformly bounded variance. AI

IMPACT This research provides theoretical insights into neural network initialization, potentially impacting the design and training of future models.

RANK_REASON The item is an academic paper published on arXiv detailing theoretical findings about neural network initialization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gaussianity Fair: The Riddle of Anomalous yet Non-Gaussian Diffusion
• Gotit.pub
• Hugging Face
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →