tokens-per-parameter ratio

LLM pretraining: Optimal domain data repetition linked to validation loss

A new research paper explores the optimal repetition of domain-specific data during the pre-training of large language models (LLMs). The study found that as models scale and require larger training-token budgets, the o…