Layer

Researchers quantify self-attention's token retrieval capacity in language models

Researchers have investigated the retrieval capacity of self-attention mechanisms in language models, aiming to understand how many tokens from a model's context are actually utilized. The study proposes a method to mea…