LLM pretraining: Optimal domain data repetition linked to validation loss

A new research paper explores the optimal repetition of domain-specific data during the pre-training of large language models (LLMs). The study found that as models scale and require larger training-token budgets, the optimal repetition count for high-quality domain data mildly increases with model size, contrary to expectations. This optimal repetition count is more strongly correlated with the final validation loss of a domain than with the amount of unique data available. The findings suggest that repetition strategies tuned on smaller models can serve as a practical estimate for larger ones. AI

IMPACT Suggests methods to improve LLM training efficiency by optimizing data repetition, potentially leading to better performance with existing compute budgets.

RANK_REASON Research paper detailing findings on LLM pretraining methodology.

Read on Hugging Face Daily Papers →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Litmaps
• ScienceCast
• Scite
• domain data
• large language models
• LLM Pretraining
• model size
• tokens-per-parameter ratio
• training-token budgets
• validation loss

• paper
• model release

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →