New analysis reveals how step size impacts SGD alignment phenomenon

This paper analyzes the phenomenon of "suspicious alignment" in stochastic gradient descent (SGD) when dealing with ill-conditioned optimization problems. The study focuses on how step size selection influences the alignment of gradient updates with dominant subspaces. Researchers propose a step-size condition that differentiates between alignment-decreasing and alignment-increasing regimes, and demonstrate that under certain conditions, projecting SGD updates to the dominant space can paradoxically increase loss. AI

IMPACT Provides a theoretical understanding of SGD behavior, potentially informing the development of more robust optimization techniques for AI models.

RANK_REASON This is a research paper published on arXiv detailing a theoretical analysis of an optimization algorithm. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• SGD
• Tianyu Pang

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →