Researchers explore negative preconditioning exponents in adaptive optimizers

Researchers have investigated the impact of negative preconditioning exponents in adaptive optimizers, specifically examining their interaction with the global learning rate. Through a controlled study across four distinct environments, they found that the optimal exponent decreases almost linearly with the logarithm of the learning rate. This suggests that negative exponents are not universally optimal but rather a regime produced by the combined action of learning rate and preconditioning, particularly at higher step sizes. The study also highlighted a conflict in model selection, where source-domain validation favors a different preconditioning regime than what maximizes robustness to environmental changes. AI

IMPACT This research could lead to more robust and generalizable adaptive optimizers for machine learning models.

RANK_REASON The cluster contains a research paper published on arXiv detailing a study on adaptive optimizers. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• cs.LG
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Litmaps
• ScienceCast
• scite Smart Citations

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →