Recurrent Neural Networks: Anti-Collapse Dynamics Enable Multi-Time-Scale Learning

A new research paper explores the challenges of long-range learning in recurrent neural networks (RNNs) trained with stochastic gradient descent. The study identifies a competition between state and parameter dynamics that leads to either a collapsed regime with rapid forgetting or an extended, anti-collapsed regime with slower, power-law forgetting. This extended regime, crucial for learning long-range dependencies, is sustained by heavy-tailed fluctuations in the learning dynamics, which act as a mechanism rather than noise to be suppressed. AI

IMPACT This research could lead to improved training methods for recurrent neural networks, enabling them to learn longer-term dependencies more effectively.

RANK_REASON The cluster contains a single academic paper detailing novel research findings in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• cs.LG
• Recurrent Neural Networks
• stochastic gradient descent

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →