New research quantifies language model recall limits

A new research paper explores the "in-context binding capacity" of language models, defining it as the number of assignments a model can recall before losing track of entity-value associations. The study measured this limit across 12 models under 3B parameters and 30 open models up to 12B parameters, finding a power-law relationship between model scale and recall capacity. The research also investigated how training recipes and direct task training influence this capacity, suggesting interference can lower measured capacity by reducing single-binding recall. AI

IMPACT Provides a new metric for evaluating language model performance, potentially guiding future model development.

RANK_REASON Research paper published on arXiv detailing a new metric for language model performance. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• 12B parameters
• 3B parameters
• arXiv
• Hugging Face
• In-Context Binding Capacity in Language Models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →