Neural models' preference for submodels explored via geometric framework

Researchers have developed a geometric framework to investigate whether neural models favor their submodels, specifically focusing on subnetworks with inactive neurons. For shallow multilayer perceptrons, subnetworks with inactive hidden neurons are never preferred. In deeper multilayer perceptrons, conditions are identified where these subnetworks are also not preferred. For self-attention models, lower-rank submodels are generally not favored, though exceptions exist where they lie on the model's boundary. The study suggests that, apart from specific cases, the preference for submodels is not explained by the neural model's geometry but rather by other factors like training dynamics. AI

IMPACT Provides a theoretical framework for understanding submodel preference, potentially informing future model optimization and interpretability research.

RANK_REASON Academic paper detailing a new geometric framework for analyzing neural models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• Litmaps
• multilayer perceptron
• Neural Models for Large-scale Semantic Role Labelling
• ScienceCast
• scite Smart Citations
• self-attention

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →