New Transformer models leverage optimization algorithms for improved performance

Researchers have developed a new family of Transformer models inspired by optimization algorithms, aiming to improve training efficiency and performance. These models, including a 'triple-momentum' variant called TMMFormer, interpret Transformer layers as steps in an optimization process. In pretraining experiments, the TMMFormer achieved the lowest validation loss, outperforming standard Transformers and demonstrating that momentum, rather than preconditioning, is the key driver of gains. The TMMFormer also exhibits flatter minima, leading to better generalization and reduced forgetting. AI

影响 Introduces novel architectural improvements for Transformers that could enhance training efficiency and model generalization.

排序理由 The cluster contains a research paper introducing a new model architecture and experimental results. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

• Momentum Streams
• TMMFormer
• Transformer

• 论文
• 模型发布

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →