FineWeb-Edu-Dedup

Data repetition significantly harms language model performance, research finds

A new research paper published on arXiv explores the detrimental effects of data repetition in language models, particularly in the era of Chinchilla-style scaling laws. The study quantifies the 'Compute-Equivalent Gain…