New research reveals slow convergence rates for Gradient Descent in neural networks

Researchers have analyzed the convergence rate of Gradient Descent (GD) in a simplified two-neuron ReLU neural network with two training instances. They theoretically proved that while GD can achieve an optimal robustness margin, its convergence rate is extremely slow, scaling as \Theta(1/\ln(t)). This finding, demonstrated through empirical simulations, establishes the first explicit lower bound on the convergence rate of the robustness margin in a non-linear model. The analysis involved a detailed examination of GD trajectories and the system's dynamics to bound the decision boundary's movement. AI

IMPACT Highlights potential limitations in training efficiency for certain neural network architectures.

RANK_REASON Academic paper detailing theoretical findings on neural network convergence. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• gradient descent
• Guy Smorodinsky
• rectifier

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →