研究人员推出了一种新颖的谱保持优化器Pion，专为训练大型语言模型而设计。与Adam等传统加性优化器不同，Pion利用正交变换更新权重矩阵，保持其奇异值和谱范数。如实证结果所示，这种方法为LLM的预训练和微调提供了一种稳定且具有竞争力的替代方案。 AI
影响 引入了一种新的优化方法，可以提高LLM的训练稳定性和性能。
排序理由 该集群包含一篇详细介绍LLM新优化技术的学术论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
Pion优化器保持谱以实现稳定的LLM训练
报道来源 [2]
-
Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
arXiv:2605.12492v1 Announce Type: cross Abstract: We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through l…
-
Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preservi…