A new research paper explores the optimal repetition of domain-specific data during the pre-training of large language models (LLMs). The study found that as models scale and require larger training-token budgets, the optimal repetition count for high-quality domain data mildly increases with model size, contrary to expectations. This optimal repetition count is more strongly correlated with the final validation loss of a domain than with the amount of unique data available. The findings suggest that repetition strategies tuned on smaller models can serve as a practical estimate for larger ones. AI
IMPACT Suggests methods to improve LLM training efficiency by optimizing data repetition, potentially leading to better performance with existing compute budgets.
RANK_REASON Research paper detailing findings on LLM pretraining methodology.
Read on Hugging Face Daily Papers →
- alphaXiv
- arXiv
- CatalyzeX
- Connected Papers
- DagsHub
- Gotit.pub
- Hugging Face
- Influence Flower
- Litmaps
- ScienceCast
- Scite
- domain data
- large language models
- LLM Pretraining
- model size
- tokens-per-parameter ratio
- training-token budgets
- validation loss
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
