Prasanna Mayilvahanan

Pretraining data dictates LLM scaling laws, study finds

Researchers have identified that the pretraining data is the primary determinant of loss-to-loss scaling laws in large language models. Their experiments indicate that factors such as model size, optimization hyperparam…