How does feature learning reshape the function space?
Researchers have precisely characterized how feature learning in neural networks reshapes the function space during gradient descent training. Their analysis, conducted in a high-dimensional proportional regime, shows that after a large gradient step, the feature distribution approximates a target-dependent spiked Gaussian covariance. This process induces a data-adaptive kernel that modifies the function space's spectral structure, selectively amplifying directions aligned with the target signal. AI
IMPACT Provides a theoretical framework for understanding how neural networks learn features, potentially guiding future model development.