Towards Engineering Scaling Laws with Pretraining Data Composition
Researchers are exploring how to engineer scaling laws for models in particle physics, drawing parallels to large language models. Unlike natural language or image domains, fundamental physics benefits from high-fidelity simulators that generate synthetic data affordably. This allows for dataset engineering to influence model scaling, favoring more data over larger parameters. For the task of classifying hadronic jets, a study demonstrated that by including diverse and task-aligned pretraining data, the scaling behavior could be shifted towards requiring more data. AI
IMPACT This research could lead to more efficient model training in scientific domains by optimizing data composition.