研究人员开发了两种新的优化技术 DMuon 和分层 Muon (HiMuon)，以提高基于矩阵正交化的优化器（如 Muon）的效率。DMuon 可集成到现有训练流程中，显著加快基础模型和大型语言模型的训练速度，将延迟接近 AdamW 水平。而 HiMuon 则采用分块方法进行 Newton-Schulz 更新，减少计算量，并实现 transformer 训练的高效 GPU 利用。此外，Tensorion 被引入作为 Muon 的张量感知泛化，将其能力扩展到高阶张量，并在计算机视觉任务中展现出潜力。 AI
影响 这些优化技术的进步可能导致大规模 AI 模型（尤其是在基础模型和计算机视觉等领域）的训练更快、更高效。
排序理由 多篇研究论文介绍了深度学习的新优化技术。
AI 生成摘要 · Google Gemini · 来自 7 个来源。 我们如何撰写摘要 →
arXiv:2606.27153v1 Announce Type: cross Abstract: Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads. The matrix-aware updates offer a compelling alternative to conven…
arXiv:2606.27216v1 Announce Type: cross Abstract: Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices. For an $H \times W$ matrix, with $r=\min\{H,W\}$ and $s=\max\{H,W\}$, $K$ step…
Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices. For an $H \times W$ matrix, with $r=\min\{H,W\}$ and $s=\max\{H,W\}$, $K$ steps of the full-matrix Newton-Schulz update require …
Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads. The matrix-aware updates offer a compelling alternative to conventional element-wise optimization, particularly as …
arXiv:2606.25975v1 Announce Type: new Abstract: Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models. Recent work has shown th…
Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models. Recent work has shown that exploiting matrix structure can improve optim…
Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models. Recent work has shown that exploiting matrix structure can improve optim…