Apple researchers unveil data mixture scaling laws for LLM pretraining

Apple Machine Learning Research has published findings on optimizing data mixtures for pretraining language models, particularly when dealing with limited target data. Their extensive study, involving over 2,000 training runs, reveals that repeating scarce target data up to 15-20 times is more effective than previously thought, especially when combined with abundant generic data. The research introduces a new scaling law that accounts for the diminishing value of repeated tokens and the regularizing effect of generic data, offering a principled method for configuring data mixtures under constraints. AI

IMPACT Provides a principled method for optimizing data mixtures in LLM pretraining, potentially improving efficiency and performance on specialized tasks.

RANK_REASON The cluster contains a research paper detailing new methods and findings in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Apple Machine Learning Research →

• Anastasiia Sedova
• Apple Inc.
• Conference on Neural Information Processing Systems
• International Conference on Machine Learning
• Natalie Schluter
• Pierre Ablin
• Skyler Seto

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →