Pro-KLShampoo optimizer improves LLM pre-training with spectral structure analysis

Researchers have developed Pro-KLShampoo, an optimization technique that combines gradient preconditioning with orthogonalization for more efficient LLM pre-training. This method leverages the observed spike-and-flat eigenvalue spectra in KL-Shampoo's preconditioners by restricting spectral structure to a tracked subspace and applying orthogonalization to the remaining directions. Pro-KLShampoo demonstrated superior performance over standard KL-Shampoo in terms of validation loss, memory usage, and training time across multiple pre-training scales, including GPT-2 and LLaMA models. AI

影响 Introduces a more efficient optimization method that could reduce compute costs for LLM pre-training.

排序理由 Academic paper introducing a novel optimization technique for LLM pre-training. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

• GPT-2
• KL-Shampoo
• LLaMA
• LLM
• Muon
• Pro-KLShampoo

• 论文
• 基础设施

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →