New LeMA framework enables scalable Levenberg-Marquardt training for large deep learning models

Researchers have developed LeMA, a distributed training framework for deep learning models that makes the Levenberg-Marquardt (LMA) algorithm practical for large-scale neural networks. LeMA addresses the prohibitive memory and computational costs associated with LMA by employing an exact dual formulation that removes memory dependence on parameter dimensions and a Jacobian sharding strategy for parallel computation. This approach allows for exact LMA steps with billion-parameter models, achieving significant speedups and high scaling efficiency on multi-GPU platforms. AI

IMPACT Enables more efficient training of large deep learning models, potentially accelerating research and development in AI.

RANK_REASON The cluster contains a research paper detailing a new algorithm and framework for deep learning training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Adam
• arXiv
• graphics processing unit
• Hugging Face
• Jacobian matrix
• LeMA

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →