Model Collapse as Cultural Evolution
Researchers have reframed the phenomenon of model collapse, where large language models degrade when trained on their own outputs, as a cultural evolution process. By applying iterated learning theory, they derived and tested five predictions using LLaMA-2-7B and Mistral-7B models across multiple languages. A key finding was that compositionality initially increases then decreases during unfiltered self-training, a pattern that persists even with regularized data and is only mitigated by task-grounded filtering. AI
IMPACT Offers a new theoretical lens for understanding and mitigating model collapse, potentially improving self-training pipeline design.