Researchers have developed a new meta-training algorithm called Efficient Long-Horizon (ELO) learning to address limitations in current learned optimizers (LOs). ELO efficiently scales meta-training to long-horizon inner problems by reallocating compute to longer failure regimes and provides stable learning signals through decoupled progressive expert supervision. This approach improves the performance and out-of-distribution generalization of LOs on downstream tasks like language modeling and image classification, with ELO-trained optimizers consistently outperforming AdamW and competing with Muon. AI
IMPACT This new meta-training algorithm could lead to more efficient and effective optimizers for AI models, potentially improving performance on complex tasks.
RANK_REASON The cluster contains a research paper detailing a new algorithm for learned optimization. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
