New HySPE positional encoding method shows superior extrapolation over RoPE

Researchers have introduced Hyperbolic Symplectic Positional Encoding (HySPE), a novel method for grounding positional attention in non-compact symplectic transformations. Unlike Rotary Position Embedding (RoPE) which uses rotations, HySPE employs a damped symmetric composition of dual shears for its hyperbolic branch. This approach offers two spectral decay rates per channel pair and ensures length-independent numerical bounds while matching RoPE's forward latency. Experiments on TinyShakespeare demonstrated HySPE's superior extrapolation capabilities, maintaining invariant perplexity significantly better than RoPE. On a Transformer model trained on WikiText-103, HySPE showed comparable in-domain performance to RoPE and a substantial reduction in tail perplexity when extrapolating to longer sequences. AI

IMPACT HySPE's demonstrated robustness in sequence length extrapolation could lead to more capable and efficient large language models.

RANK_REASON The cluster contains a research paper detailing a new method for positional encoding in NLP models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Hugging Face
• Hyperbolic Symplectic Positional Encoding
• Rope
• Rotary Position Embedding
• RTX 4090
• Symplectic (UK)
• TinyShakespeare
• WikiText-103

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →