New research details learning-rate schedules for least-squares regression

A new research paper published on arXiv analyzes the asymptotic risk of learning-rate schedules for least-squares regression with discretely distributed features. The study focuses on the last iterate of stochastic gradient descent (SGD) in Slock's model, deriving exact finite-time risk and asymptotic equivalents for constant and Warmup-Stable-Decay (WSD) schedules. The findings indicate that a tuned base learning-rate leads to a scaling law decay of $\Theta(T^{-q})$, where any cooldown of fixed relative length improves risk by a polynomial factor, with linear decay minimizing the constant. AI

IMPACT Provides theoretical insights into optimization algorithms relevant to machine learning model training.

RANK_REASON The item is an academic paper published on arXiv detailing theoretical findings in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• alphaXiv
• arXiv
• Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features
• CatalyzeX Code Finder for Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• Influence Flower
• machine learning
• ScienceCast
• Slock's model
• stochastic gradient descent
• Warmup-Stable-Decay

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →