M+Adam optimizer improves low-precision LLM training

Researchers have introduced M+Adam, a novel optimization method designed to improve the accuracy of training large language models with low-precision weights. Standard optimizers can struggle with low precision, leading to stalled progress, especially at large weight magnitudes. M+Adam addresses this by combining both additive and multiplicative update types, leveraging their complementary strengths to ensure consistent progress across various weight magnitudes and sign changes. Experiments with LLaMA-style models and different precision levels, including BF16, FP8, and FP4, demonstrate that M+Adam consistently enhances low-precision training outcomes. AI

IMPACT M+Adam could enable more efficient training of large models by reducing the precision requirements, potentially lowering computational costs and hardware demands.

RANK_REASON The cluster contains a research paper detailing a new optimization method for training machine learning models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• bfloat16
• Chinchilla
• Fp8
• Hugging Face
• LLaMA
• M+Adam

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →