Researchers have developed a new method for scheduling high-quality data during large language model (LLM) training, addressing the scarcity of such data. The approach, termed Drop-Stable-Rampup, extends functional scaling laws to incorporate data quality, revealing two distinct regimes for data utilization. In the noise-limited regime, high-quality data acts as a signal amplifier by lowering batch sizes, while in the signal-limited regime, it functions as a noise suppressor through late placement. Experiments on a 15B Mixture-of-Experts model demonstrated significant accuracy improvements, particularly in mathematical reasoning tasks, compared to existing methods. AI
IMPACT Optimizes the use of scarce high-quality data in LLM training, potentially leading to more accurate models, especially in complex reasoning tasks.
RANK_REASON The cluster contains an academic paper detailing a new method for LLM training.
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
