New research highlights gradient flow vs. SGD discrepancies in neural networks

A new paper published on arXiv explores the discrepancies between population gradient flow and finite-batch stochastic gradient descent (SGD) in neural network adaptation. The research identifies a specific mechanism where gradient flow can mispredict SGD's behavior, particularly in scenarios involving ReLU activation units. This divergence becomes significant after extensive pretraining, where small-step SGD may fail to recover, unlike gradient flow, due to the joint limits of small steps and long training durations. The study suggests that the instability is concentrated on specific inputs where ReLU gates disagree, and the probability of recovery is linked to the disagreement budget, batch size, and learning rate. AI

IMPACT Highlights potential limitations in understanding neural network adaptation through gradient flow approximations, impacting theoretical research.

RANK_REASON Academic paper published on arXiv detailing a theoretical finding in neural network training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• ReLU
• SGD

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →