Researchers have developed CuraWeb, a new 2 trillion token English corpus designed to improve the pretraining data for large language models. Unlike previous methods that focused on singular optimization objectives, CuraWeb employs a novel framework for joint optimization of quality, redundancy, and diversity. This approach utilizes dual-track cleaning and hybrid deduplication techniques, balanced by a multi-objective sampler. Experiments show that CuraWeb, when applied to Common Crawl data, significantly outperforms existing datasets, yielding an average performance gain of 1.8% across various benchmarks, particularly in knowledge-intensive and reasoning tasks. AI
IMPACT Establishes a new standard for web-scale data curation, potentially improving LLM performance on knowledge-intensive and reasoning tasks.
RANK_REASON Research paper detailing a new method for curating pretraining data for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
