New SGDIR analysis yields dimension-free risk bounds

Researchers have analyzed a variant of stochastic gradient descent with initial regularization (SGDIR), deriving dimension-free upper bounds on its expected excess risk for the squared loss. In noiseless scenarios, new bounds were established for both averaged and non-averaged SGDIR under various assumptions, with some bounds reaching $m^{-3+\epsilon}$ order. The study also presents a lower bound that closely matches the upper bounds in specific regimes and includes an instance-based comparison between SGDIR and ridge regression in noisy conditions, showing SGDIR's risk is comparable. Numerical experiments on both synthetic and real data support these theoretical findings. AI

IMPACT Provides theoretical insights into optimization algorithms relevant to machine learning model training.

RANK_REASON The cluster contains an academic paper detailing a new theoretical analysis of an algorithm. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• arXiv
• Stochastic gradient descent with initial regularization

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →