A new paper explores how population gradient flow, a common method for analyzing neural network adaptation, can inaccurately predict the behavior of finite-batch stochastic gradient descent (SGD). The research identifies a specific mechanism where the discrepancy arises, particularly in a two-unit ReLU regression task. The study demonstrates that while gradient flow recovers on a target task after a certain pretraining period, online SGD with finite batches can fail with high probability under specific conditions related to batch size, step size, and pretraining duration. This failure is linked to disagreements between ReLU gates on certain inputs, which are then exacerbated by weight decay. AI
IMPACT This research highlights potential limitations in current theoretical models for neural network training, suggesting a need for more robust methods to predict and understand adaptation dynamics.
RANK_REASON The cluster contains an academic paper detailing a novel finding in neural network training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
