Researchers have developed a new theoretical framework to analyze the early training dynamics of large nonlinear residual networks. Their model, which considers a sequential limit where width, depth, data points, and input dimension all diverge, reveals that in high input dimensions, network outputs are restricted to linear functions of inputs early in training. This finding suggests that standard mean-field theories for residual networks may be insufficient for understanding feature learning in such regimes. The study also investigates methods for transferring learning rate hyperparameters across different dataset sizes and input dimensions, with numerical simulations showing strong agreement with the theoretical predictions. AI
IMPACT Suggests limitations in current mean-field theories for understanding feature learning in large residual networks, potentially guiding future research.
RANK_REASON Academic paper published on arXiv detailing theoretical findings about neural network training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
