Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
A new research paper explores the relationship between model flatness and generalization in neural networks. Despite prior work suggesting symmetries render flatness a vacuous metric, this study demonstrates a connection for learning multi-index models with homogeneous neural networks. The research identifies specific classes of non-generalizing interpolators and proves that the "flattest" interpolators achieve low population loss, establishing a direct link between flatness and generalization across various activations and data distributions. AI
IMPACT Establishes a theoretical link between model flatness and generalization, potentially guiding future research in neural network optimization and design.