attention weights
PulseAugur coverage of attention weights — every cluster mentioning attention weights across labs, papers, and developer communities, ranked by signal.
-
Researchers quantify self-attention's token retrieval capacity in language models
Researchers have investigated the retrieval capacity of self-attention mechanisms in language models, aiming to understand how many tokens from a model's context are actually utilized. The study proposes a method to mea…
-
New theory suggests language needs two parameters beyond word co-occurrence
A new research paper proposes that language understanding relies on two parameters: amplitude (word co-occurrence) and phase (how meanings combine). The authors argue that current Transformer models lack an explicit rep…
-
ALiBi positional encoding numerical failure identified in AI models
Researchers have identified a significant numerical failure in ALiBi positional encodings, a component used in many state-of-the-art pretrained models. The linear bias scaling in ALiBi can underflow floating-point preci…