New theory links correlation propagation to Neural Tangent Kernel in deep learning

Researchers have established a theoretical link between correlation propagation and the Neural Tangent Kernel (NTK) in deep neural networks. By combining mean-field and random matrix theories, they demonstrated that correlation propagation to infinite depth is only possible at a critical point in the weight-bias variance plane. At this critical point, the end-to-end Jacobian vanishes algebraically with depth, leading to the NTK becoming proportional to the output correlation. The study also showed that orthogonal initialization better controls asymptotic dynamics in finite-width, finite-depth networks compared to Gaussian initialization. AI

IMPACT Provides a theoretical framework for understanding deep learning initialization and its impact on information propagation and learning dynamics.

RANK_REASON Academic paper detailing theoretical advancements in deep learning dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Gaussian initialisation
• Neural tangent kernel
• orthogonal initialisation

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →