Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
Researchers have identified a key issue in scaling up AI model training data mixtures, termed "repetition mismatch." This occurs when the optimal data mixture changes as training budgets increase due to the varying repetition rates of high-quality, limited datasets. A new subsampling procedure that matches the target repetition rate can accurately predict optimal mixtures from significantly smaller experiments, improving efficiency and accuracy. AI
IMPACT Improves efficiency and accuracy in training large AI models by addressing data mixture scaling issues.