Classifier quality filtering vulnerable to Wikipedia-style reformatting

Researchers have identified a significant vulnerability in classifier-based quality filtering, a common technique for curating pre-training data for large language models. Their study demonstrates that simple reformatting of content, mimicking Wikipedia's style, can trick these classifiers into misjudging document quality. This could lead to the inclusion of lower-quality data in training corpora, potentially impacting model performance. AI

IMPACT Highlights a potential flaw in data curation for LLMs, which could impact model quality if not addressed.

RANK_REASON Academic paper detailing a new finding about a common AI technique. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Classifier-based Quality Filtering
• FineWeb-Edu CQF model
• Mateusz Klimaszewski
• Wikipedia

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →