研究人员提出了一种新的深度学习优化器设计原则，该原则与神经网络架构的内在对称性相符。与目前逐坐标操作参数的Adam等优化器不同，所提出的对称性兼容优化器被设计成对不同权重块的特定对称群具有等变性。这种方法已应用于嵌入层、LM头、SwiGLU MLP和MoE路由器等各种组件，产生了新颖的更新规则。在语言模型上的实验表明，与标准的AdamW相比，这些新型优化器在验证损失和训练稳定性方面持续得到改善。 AI
影响 引入了新颖的优化器设计，改善了语言模型的训练稳定性和最终验证损失。
排序理由 该集群包含一篇学术论文，详细介绍了深度学习优化器设计的新理论原则和实验验证。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2605.18106v1 Announce Type: cross Abstract: A striking geometric disparity has long persisted in the practice of deep learning. While modern neural network architectures naturally exhibit rich symmetry and equivariance properties, popular optimizers such as Adam and its var…
A striking geometric disparity has long persisted in the practice of deep learning. While modern neural network architectures naturally exhibit rich symmetry and equivariance properties, popular optimizers such as Adam and its variants operate inherently coordinate-wise, renderin…