New framework precisely models multi-class SGD dynamics in high dimensions

Researchers have developed a new framework to analyze the learning dynamics of multi-class stochastic gradient descent (SGD) in high-dimensional settings. The framework provides exact expressions for key metrics like risk and signal overlap, expressed as a deterministic system of ordinary differential equations in the high-dimensional limit. This approach is applicable to a wide range of optimization problems and can accommodate scenarios where the number of classes increases with dimension. The study details the impact of data anisotropy on binary logistic regression and least-squares loss, identifying a learning-rate threshold for the latter and a structural phase transition for the former, particularly in models with sparse covariance matrices. AI

IMPACT Provides a theoretical foundation for understanding and potentially improving the training of complex, multi-class machine learning models.

RANK_REASON Academic paper detailing a new theoretical framework for analyzing machine learning dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• arXiv
• Elizabeth Collins-Woodfin
• least squares method
• logistic regression model
• stochastic gradient descent

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →