New research models SGD dynamics as percolation process

Researchers have modeled the dynamics of Stochastic Gradient Descent (SGD) as a percolation process, revealing how architectural symmetries cause subnetworks to merge in discrete blocks. These transitions manifest as variance spikes in a macroscopic order parameter, similar to physical phase transitions. The study demonstrates that this trapping mechanism and its associated scaling cascade also apply to Adam and AdamW optimizers under a heavy-tailed noise model. AI

IMPACT Provides a new theoretical framework for understanding optimization dynamics in deep learning.

RANK_REASON The cluster contains a single academic paper published on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Adam
• AdamW
• arXiv
• DagsHub
• Hugging Face
• Sai Niranjan Ramachandran
• Stochastic Gradient Descent

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →