nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding
Researchers have introduced nD-RoPE, a novel method for generalizing Rotary Position Embedding (RoPE) to n-dimensional spaces. Unlike previous approaches that treated dimensions independently, nD-RoPE formulates position and frequency as coupled n-dimensional vectors. This unified theoretical framework, derived from a translation-invariant perspective, allows for richer cross-dimensional interactions. Experiments on image, video, and point cloud data show that nD-RoPE consistently improves performance and generalization in high-dimensional settings. AI
IMPACT This new method could enhance the performance of Transformer models in processing high-dimensional data like images and videos.