Researchers have explored the effects of recursive self-training on language models, specifically fine-tuning GPT-2 on its own output over multiple generations. They observed that as the training temperature increases, different measures of information entropy, such as min-entropy and collision entropy, transition from loss to gain at distinct temperature points. This process leads to a reduction in the number of unique word types while simultaneously decreasing the dominance of the most frequent words. The study found this ordering of entropy changes to be consistent across various settings, including different measurement samplers, word budgets, and even when fine-tuning on human text, suggesting that entropy profiles should be reported alongside independent quality measures. AI
IMPACT This research provides insights into how self-training affects language model vocabulary and word dominance, potentially informing future training methodologies.
RANK_REASON The cluster contains an academic paper detailing novel research findings on language model training. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
