What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
Researchers have developed a unified spectral analysis framework to understand knowledge transfer in machine learning, particularly in high-dimensional linear regression. This framework explains how knowledge distillation and weak-to-strong generalization work by identifying two key mechanisms: spectral horizon expansion and spectral denoising. The study suggests that the effectiveness of knowledge transfer depends on the interaction between implicit regularization and varying spectral learning speeds. AI
IMPACT Provides a unified theoretical lens for understanding knowledge transfer mechanisms, potentially guiding future model development and training strategies.