Language Models Show Position-Dependent Repetition Effects

A new research paper titled "When More Becomes Less: Position-Dependent Repetition Effects in Language Models" has been published on arXiv. The study reveals that the frequency of a target token's repetition impacts its prediction differently based on its position within the text. Specifically, when a readout slot is placed immediately after repeated tokens, the prediction probability increases with repetition. However, when the readout slot is within a new sentence, the probability follows an inverted-U pattern, peaking early and then declining with more repetitions. This effect was observed across 13 different language models and replicated in Spanish, Chinese, German, and French. AI

IMPACT Reveals a nuanced understanding of how language models process repeated information, potentially impacting future model design and evaluation.

RANK_REASON Research paper published on arXiv detailing findings about language model behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• French
• German
• Hugging Face
• Language Models
• Spanish
• Standard Chinese
• When More Becomes Less: Position-Dependent Repetition Effects in Language Models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →