Researchers have identified a key reason why certain optimizers like Muon outperform Adam when training equivariant neural networks. The issue stems from how Adam handles learning rates across different blocks within an equivariant linear layer. Specifically, Adam rescales weights individually without respecting the boundaries of these blocks, leading to inconsistent step sizes. The proposed solution involves normalizing each block's update separately, which, when combined with tuned momentum coefficients, makes Adam competitive with Muon on various datasets. AI
IMPACT Identifies a specific training challenge in equivariant networks, potentially leading to more efficient model development.
RANK_REASON Academic paper detailing a novel finding about neural network training. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
