Researchers have developed a theoretical framework, termed a "Conditional No-Hair Theorem," to explain how large language models forget their prompts over time. The study found that the influence of a prompt on a transformer model decays as a power law, with trained models forgetting faster than untrained ones. This phenomenon is attributed to the dilution of prompt influence through attention mechanisms and the growth of generated text, with a theoretical bound suggesting prompt influence vanishes asymptotically conditional on the generated text. A toolkit for measuring this forgetting rate in Hugging Face models is also provided. AI
IMPACT Provides a theoretical understanding of prompt decay in LLMs, potentially informing future model design and prompting strategies.
RANK_REASON Academic paper detailing a theoretical finding about LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →