Researchers have developed SoftServe, a novel family of Quasi-Newton (QN) methods designed to address the challenges of non-convexity and massive parameter sizes in deep learning. Unlike traditional QN methods, SoftServe derives positive-definite curvature estimates even in negative curvature regions and scales to large neural networks through diagonal and Kronecker-factored variants. The method utilizes the coupled Newton-Schulz iteration for matrix operations, replacing expensive decompositions with GPU-friendly matrix multiplications. SoftServe has demonstrated superior performance on ill-conditioned problems, including physics-informed neural networks and diffusion models, achieving lower losses than established optimizers like Adam and Muon. AI
IMPACT This new optimization method could lead to more efficient training of large and complex deep learning models, potentially accelerating research and development in areas like physics-informed models.
RANK_REASON The cluster contains a research paper detailing a new method for deep learning optimization. [lever_c_demoted from research: ic=1 ai=1.0]
- Adam
- alphaXiv
- arXiv
- Berglund
- CatalyzeX
- DagsHub
- Gotit.pub
- Hugging Face
- IArxiv
- muon
- ScienceCast
- SoftServe
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
