NTK scaling

Transformer models gain absolute position awareness from causal mask and residual stream

Researchers have identified two key architectural components in decoder-only Transformers that contribute to the model's ability to distinguish absolute position, despite positional encoding methods like RoPE primarily …