Ori optimizer narrows LLM training gap with drift-aware k-PCA

Researchers have developed Ori, a new optimization technique for training large language models (LLMs) that addresses the challenge of low-rank training. Ori uses a drift-aware streaming k-PCA algorithm, inspired by Oja++, to continuously track shifting principal gradient directions. This approach significantly reduces the performance gap between low-rank and full-rank LLM training, showing up to a 69.3% improvement on a LLaMA pretraining task and outperforming other methods on Roberta-Base fine-tuning tasks. AI

IMPACT This research could lead to more efficient and accurate training of large language models, potentially reducing computational costs and improving performance on various NLP tasks.

RANK_REASON The cluster contains an academic paper detailing a new method for LLM training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• LLaMA
• LLM
• Oja++

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →