Jelinek-Mercer

Transformer models use Jelinek-Mercer and Dirichlet-style smoothing for in-context learning

Researchers have identified two complementary smoothing mechanisms within transformer models that are believed to underlie in-context learning. The first mechanism, observed at a finite attention-weight scale, acts as a…