Recurrent Neural Network Learning Dynamics Near Bifurcations Analyzed

A new research paper explores the dynamics of learning in recurrent neural networks (RNNs) near critical transition points, known as bifurcations. The study utilizes the global empirical Neural Tangent Kernel (GeNTK) to demonstrate that the learning geometry becomes amplified and anisotropic, concentrating towards specific low-rank channels. This theoretical framework is validated through experiments with high-dimensional RNNs and a multi-task LeakyRNN, showing correlations between GeNTK amplification and abrupt loss changes, subtask interference, and shifts in internal dynamics. AI

IMPACT Provides a theoretical framework for understanding and diagnosing learning behavior in complex neural networks near critical transitions.

RANK_REASON Academic paper detailing theoretical and experimental analysis of learning dynamics in RNNs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Center-Manifold Reduction of Learning at Bifurcations
• Fisher Information Matrix
• Gentkowski
• Interference and Rich Learning in Recurrent Neural Networks
• James Hazelden
• LeakyRNN
• MemoryPro
• Neural tangent kernel
• recurrent neural network

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →