Researchers have developed LinMix, a novel method for optimizing the mixture of domains in pretraining data for language models. Unlike traditional approaches that rely on extensive proxy model training and regression, LinMix frames the problem as domain attribution. It employs a hyperparameter-free vertex sampling strategy on a capped simplex, which decorrelates domain effects and enables a linear model to directly identify domains with positive contributions. Experiments with OLMo2 and Qwen 3.5 models demonstrated that LinMix significantly outperforms existing methods on benchmarks like MBPP and GSM8K, achieving an 11% average improvement in accuracy over nonlinear baselines. AI
IMPACT Optimizes pretraining data mixtures, potentially leading to more efficient and accurate language models across various benchmarks.
RANK_REASON The cluster contains a research paper detailing a new method for data mixing in language model pretraining. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
