New adaptive batch sizing method cuts training steps by up to 66%

Researchers have developed a new method for adaptive batch sizing in machine learning that accounts for the non-Euclidean geometry of optimizers like signSGD and spectral descent. This approach, which estimates non-Euclidean gradient noise scales using local mini-batch gradients, can significantly reduce training steps. Experiments showed up to a 66% reduction in training steps for a 160 million parameter Llama model using signSGD and spectral descent, while matching the validation loss of constant-batch baselines. AI

IMPACT This method could lead to more efficient training of large language models, reducing computational costs and time.

RANK_REASON The cluster contains a research paper detailing a new method for optimizing machine learning training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• llama
• muon
• SGD
• Shagun Gupta
• SignSGD
• Signum
• specSGD

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →