A new research paper explores the impact of AI-generated text on language model pretraining, finding that while initially beneficial, excessive AI text can harm performance on human text. The study, which involved pretraining 800 language models, proposes a new scaling law that accounts for both the benefits and harms of AI tokens. The researchers recommend filtering AI text when the goal is to optimize for human text performance and suggest repeating human text before incorporating AI-generated content. AI
IMPACT This research suggests that the quality and quantity of AI-generated text used in pretraining are critical factors that could impact future language model performance.
RANK_REASON Research paper published on arXiv detailing findings about AI-generated text in pretraining data.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →