English(EN)Silver Rate Is (Almost) Optimal for Gradient Descent Acceleration
新研究推进机器学习随机优化 · 跟踪5个来源
作者PulseAugur 编辑部·[6 个来源]·
几篇近期研究论文探讨了随机优化技术的进展，特别关注梯度下降及其变体在复杂机器学习问题中的应用。其中一篇论文表明，即使存在重尾噪声，标准的随机梯度下降上升（SGDA）也能收敛，而无需梯度裁剪，并为正则化问题引入了新的无裁剪算法。另一项研究通过关注梯度映射，在约束凸凹极小极大优化方面取得了近乎最优的收敛率。此外，研究还探讨了平滑凸目标函数的随机梯度下降（SGD）的精确收敛率，确立了理论极限，并分析了使用预定步长的梯度下降加速的最优性。最后，一篇论文详细描述了常数步长随机逼近的特性，提供了有限时间收敛保证和高阶定量高斯近似。
AI
arXiv:2609.06064v1 Announce Type: cross Abstract: Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical studies mainly rely on the bounded variance assumption for stochastic gradients. Und…
arXiv:2609.08380v1 Announce Type: cross Abstract: We study the stochastic first-order oracle complexity for constrained or regularized convex-concave min-max optimization and stochastic monotone variational inequalities. We focus on the case when suboptimality is measured in term…
arXiv cs.LG
TIER_1English(EN)·Ruijie Li, Kang Chen, Tianyu Wang·
arXiv:2609.08537v1 Announce Type: cross Abstract: We study the time-uniform convergence of the raw iterate of standard stochastic gradient descent (SGD) for unconstrained smooth convex objectives. We prove that, under standard noise assumptions, the time-uniform convergence rate …
arXiv cs.LG
TIER_1English(EN)·Yuhan Ye, Kaizhao Liu·
arXiv:2609.09152v1 Announce Type: cross Abstract: We study how far gradient descent (GD) can be accelerated by predetermined nonnegative stepsizes in smooth convex optimization. Writing $p_{\mathrm{sil}}=\log_2(1+\sqrt{2})$, we prove an $\Omega\left(n^{-p_{\mathrm{sil}}-O(\sqrt{\…
arXiv:2602.13960v2 Announce Type: replace Abstract: Constant-stepsize stochastic approximation (SA) is widely used in learning for computational efficiency, yet the distribution of the iterates is typically intractable. Classical asymptotics results give $X_k^{(\alpha)} \approx X…
arXiv:2604.13130v2 Announce Type: replace-cross Abstract: We study learning to learn through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and reg…
