arXiv paper: Width and Depth Limits Commute During Training

Researchers have demonstrated that the infinite-width and infinite-depth limits of residual networks commute during the training process, not just at initialization. This commutativity holds for linear residual networks under specific scaling conditions, including the Depth-"mu"P scaling and the $\mu$P width parameterization. The study proves that for a fixed number of gradient-descent steps with gradient clipping, the network output converges in L2 as both width and depth tend to infinity, extending the commutativity result throughout any finite training horizon. AI

IMPACT Provides theoretical insights into the behavior of large neural networks during training, potentially influencing future model architectures.

RANK_REASON Academic paper detailing theoretical advancements in neural network training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• arXivLabs
• CatalyzeX Code Finder for Papers
• CORE Recommender
• DagsHub
• Depth-"mu"P
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• Influence Flower
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →