Researchers have developed a new meta-training algorithm called ELO (Efficient Long-hOrizon) to improve learned optimizers (LOs). ELO addresses the challenges of scaling meta-training to long-horizon problems and competing with established optimizers like Adam and Muon. The algorithm reallocates compute to longer failure regimes and uses progressive expert supervision for stable learning signals. Empirical studies show ELO significantly enhances LO performance on downstream language modeling and image classification tasks, with ELO-Celo2 consistently outperforming AdamW and remaining competitive with Muon on language modeling, all while requiring minimal GPU hours for meta-training. AI
IMPACT This research could lead to more efficient and effective optimizers for training large AI models, potentially reducing computational costs and improving performance on complex tasks.
RANK_REASON Academic paper detailing a new algorithm for learned optimization. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
