New theory links shock-wave dynamics to neural network training

Researchers have established a mathematical connection between shock-wave theory and the learning dynamics of stochastic gradient descent in artificial neural networks. By applying principles from differential geometry, Lie group theory, and fluid mechanics, they demonstrated that the effective dynamics of these networks can be described by a viscous Hamilton--Jacobi equation on a quotient manifold. Furthermore, the coarse-grained loss function's gradient follows a Burgers-type equation, indicating that shock formation is rigorously possible. This framework has been applied to various architectures, including multilayer perceptrons, convolutional neural networks, Transformers, and mean-field networks, suggesting potential for new diagnostics in deep learning. AI

IMPACT This theoretical framework could lead to novel diagnostics for monitoring and controlling deep learning training phases.

RANK_REASON The item is an academic paper detailing theoretical research on artificial neural networks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• artificial neural network
• Burgers-type equation
• convolutional neural network
• deep learning
• differential geometry
• fluid mechanics
• Hamilton--Jacobi Equations and Distance Functions on Riemannian Manifolds
• Lie group theory of the bessel equation of the first kind of integral order
• Mean-field Networks
• multilayer perceptron
• Shock-wave theory for rupture of rubber
• stochastic gradient descent
• transformers

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →