Layerwise LQR framework optimizes deep networks using geometry-aware control

Researchers have developed Layerwise LQR (LLQR), a new optimization framework for deep learning models. LLQR reformulates second-order optimization methods, like Newton's method, as a linear quadratic regulator problem. This approach allows for the learning of structured inverse preconditioners that capture global layerwise dynamics without computing the full curvature matrix. Experiments on ResNets and Transformers indicate that LLQR can enhance optimization speed and final model performance with minimal computational overhead. AI

IMPACT Introduces a novel optimization technique that could improve training efficiency and performance for deep learning models.

RANK_REASON Academic paper introducing a novel optimization framework for deep learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Layerwise LQR
• LLQR
• Newton
• Shampoo
• Simon Dufort-Labbé
• Transformers
• K-FAC

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →