KL-Shampoo

Pro-KLShampoo optimizer improves LLM pre-training with spectral structure analysis

Researchers have developed Pro-KLShampoo, an optimization technique that combines gradient preconditioning with orthogonalization for more efficient LLM pre-training. This method leverages the observed spike-and-flat ei…