Paper explains why relative positional encodings improve transformer generalization

A new paper proposes an optimization-based explanation for why transformers with relative positional encodings generalize better to longer sequences than those with absolute encodings. The research suggests that the implicit bias of gradient descent favors solutions that are equivariant to relative positions when using Rotary Encodings, leading to consistent performance across different sequence lengths. In contrast, absolute encodings tend to memorize specific positions, hindering extrapolation. The study confirms these findings through experiments on a minimal retrieval task and a full-sequence length-generalization task, also noting that linear attention mechanisms do not exhibit the same generalization behavior without softmax normalization. AI

IMPACT Provides a theoretical framework for understanding and potentially improving sequence length generalization in transformer architectures.

RANK_REASON The cluster contains a single academic paper detailing a new theoretical explanation for a known phenomenon in transformer models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Absolute Encodings
• gradient descent
• linear attention
• RASP-L conjecture
• Rotary Encodings
• Softmax
• transformers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →