Seesaw method accelerates LLM training by optimizing batch size and learning rate

Researchers have developed a new method called Seesaw to accelerate the training of large language models by optimizing the scheduling of batch sizes and learning rates. This approach theoretically demonstrates an equivalence between learning-rate decay and batch-size ramp-up for SGD and extends this to a proxy for Adam. Empirically, Seesaw has shown to match cosine decay in terms of FLOPs while reducing training time by approximately 36% for models trained at Chinchilla scale. AI

IMPACT This method could significantly reduce the time and computational resources required for pretraining large language models.

RANK_REASON The cluster contains an academic paper detailing a new method for accelerating LLM training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• Adam
• Alexandru Meterez
• arXiv
• Chinchilla
• Hugging Face
• Seesaw
• SGD

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →