SGD breaks scale symmetries in linear autoencoders, favoring large decoder weights

Researchers have identified a phenomenon in undercomplete linear autoencoders where finite-stepsize stochastic gradient descent (SGD) breaks scale symmetries. This process favors large decoder weights on the principal component analysis (PCA) solution manifold, leading to directed scale drift. While this drift is analytically tractable, it eventually encounters a stability boundary, resulting in solutions with sharper characteristics according to certain measures, though other sharpness metrics may move in opposing directions. AI

IMPACT Provides a theoretical understanding of how SGD influences model geometry, potentially informing future model design and training.

RANK_REASON Academic paper detailing a novel finding in machine learning theory. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Farhad Pashakhanloo
• principal component analysis
• stochastic gradient descent

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →