New MetaAdamW optimizer uses self-attention for adaptive learning rates

Researchers have developed MetaAdamW, a novel optimizer that enhances adaptive learning rates and weight decay by employing a self-attention mechanism. This Transformer-based approach dynamically adjusts hyperparameters for different parameter groups based on statistical features, aiming to overcome the limitations of uniform settings in optimizers like AdamW. Experiments across diverse tasks show MetaAdamW consistently outperforms AdamW, reducing training time or improving performance. AI

影响 Introduces a novel optimizer that could improve training efficiency and performance across various machine learning tasks.

排序理由 This is a research paper detailing a new optimization algorithm for machine learning models. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

• AdamW
• CIFAR-10
• IMDB
• MetaAdamW
• Transformer
• WikiText-2

• 论文
• 其他

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →