Transformer models use Jelinek-Mercer and Dirichlet-style smoothing for in-context learning

Researchers have identified two complementary smoothing mechanisms within transformer models that are believed to underlie in-context learning. The first mechanism, observed at a finite attention-weight scale, acts as a soft context-matching estimator, interpolating across context orders similarly to Jelinek-Mercer smoothing. The second mechanism involves a beginning-of-sequence token that introduces additive pseudo-counts, akin to Dirichlet-style smoothing. A specially constructed transformer model demonstrated these mechanisms, showing that transformers learn to regularize in-context estimation rather than merely counting, and can match or outperform classical statistical baselines. AI

IMPACT Provides a deeper mechanistic understanding of in-context learning in transformers, potentially guiding future model architectures.

RANK_REASON Academic paper detailing novel findings on transformer model mechanisms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Dirichlet
• Francesco D'Angelo
• Gotit.pub
• Hugging Face
• IArxiv
• Influence Flower
• Jelinek-Mercer
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →