Pro-KLShampoo optimizer improves LLM pre-training with spectral structure analysis

Researchers have developed Pro-KLShampoo, an optimization technique that combines gradient preconditioning with orthogonalization for more efficient LLM pre-training. This method leverages the observed spike-and-flat eigenvalue spectra in KL-Shampoo's preconditioners by restricting spectral structure to a tracked subspace and applying orthogonalization to the remaining directions. Pro-KLShampoo demonstrated superior performance over standard KL-Shampoo in terms of validation loss, memory usage, and training time across multiple pre-training scales, including GPT-2 and LLaMA models. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Introduces a more efficient optimization method that could reduce compute costs for LLM pre-training.

RANK_REASON Academic paper introducing a novel optimization technique for LLM pre-training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Pro-KLShampoo
• KL-Shampoo
• LLM
• Muon
• GPT-2
• LLaMA

• paper
• infra