Robustness of Similarity-based Positional Encoding Under Rotations: Theoretical Analysis and Experimental Validation
Researchers have theoretically analyzed and experimentally validated the robustness of similarity-based positional encoding (simPE) under rotations, a crucial aspect for applications like medical imaging. The study demonstrates that while simPE is not inherently rotation-invariant, it exhibits stability under rotational perturbations with explicit bounds. Experiments on synthetic and benchmark datasets show simPE consistently outperforms standard learned positional encoding in accuracy and other metrics when dealing with rotated images. AI
IMPACT This research provides theoretical guarantees and empirical evidence for the robustness of similarity-based positional encoding in Transformer architectures, potentially improving performance in applications sensitive to geometric transformations.