New learned optimizer improves training for BERT-Tiny, GPT-Tiny, and Vision Transformers

Researchers have developed a novel learned optimizer that dynamically recombines gradient history to improve model training efficiency and performance. This optimizer, with only 37,000 parameters, achieved significant improvements in just under an hour of GPU time. It demonstrated a 9.1% and 0.4% reduction in validation loss for BERT-Tiny and GPT-Tiny respectively, and outperformed Adam by 3.5 percentage points on a Vision Transformer and 2.7 percentage points on average across nine graph models. AI

IMPACT This new optimization technique could lead to faster and more efficient training of smaller language and vision models, potentially reducing computational costs.

RANK_REASON The cluster contains a research paper detailing a new method for optimizing machine learning models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• arXiv
• BERT-Tiny
• CatalyzeX
• DagsHub
• Gotit.pub
• GPT-Tiny
• Hugging Face
• IArxiv
• vision transformer

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →