Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
Two new research papers explore the necessity of positional encoding (PE) in transformer models. One paper demonstrates that sliding-window transformers can achieve Turing completeness without PE, suggesting that the window mechanism itself provides sufficient positional information. The other paper investigates PE's role in Vision Transformers (ViTs), finding that while ViTs can develop spatial structure without PE, PEs anchor this structure and significantly improve robustness against content-disrupting distribution shifts. AI
IMPACT Challenges the necessity of positional encodings, potentially simplifying future transformer architectures and improving efficiency.