Decoder-Only Transformers
PulseAugur coverage of Decoder-Only Transformers — every cluster mentioning Decoder-Only Transformers across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New benchmark tests transformer routing capabilities
Researchers have developed ROUTEBENCH, a new diagnostic benchmark designed to evaluate whether transformers can effectively route their in-context learning capabilities to different inductive biases. The benchmark featu…
-
New IG-Lens method precisely attributes token probability across transformer layers
Researchers have developed IG-Lens, a novel method for precisely attributing the probability of a predicted token to specific layers within decoder-only transformer models. Unlike existing tools that offer approximate o…
-
Transformer models gain absolute position awareness from causal mask and residual stream
Researchers have identified two key architectural components in decoder-only Transformers that contribute to the model's ability to distinguish absolute position, despite positional encoding methods like RoPE primarily …
-
Researchers explore efficient transformers via attention control and algorithmic capture
Researchers are exploring methods to enhance transformer efficiency and understanding. One paper introduces Budgeted Attention Allocation, a head-gating mechanism that allows for cost-quality trade-offs. Another study d…