New blockwise optimizer enhances cubic Newton methods for large-scale neural networks

This paper introduces a novel blockwise optimizer designed to improve the efficiency of cubic regularized Newton methods for large-scale neural network training. The proposed method handles arbitrarily large parameter tensors using a matrix-free approach within a Lanczos-built Krylov subspace, ensuring the step accurately minimizes the cubic model. Evaluated against existing methods like ARC, Adam, and L-BFGS, the new variants demonstrate superior performance, particularly on a 91.4M-parameter implicit neural representation, with one variant achieving a significantly higher peak signal-to-noise ratio in image fitting tasks. AI

IMPACT Introduces a more efficient optimization technique for training large neural networks, potentially accelerating research and development in areas like neural fields.

RANK_REASON The cluster contains an academic paper detailing a new optimization method for machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• ARC- 0 9_1
• Cubic regularized Newton methods
• FINER
• Limited-memory BFGS
• neural field
• Rodion Podorozhny

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →