Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
Researchers have introduced Morlet Positional Encoding (MoPE) as a novel framework for Transformer positional encoding, moving beyond traditional sinusoidal and rotary methods. MoPE utilizes the Morlet wavelet to simultaneously encode position and frequency, allowing each embedding dimension to learn its own locality bandwidth. This approach theoretically unifies existing methods and empirically shows improvements in tasks like language modeling, outperforming standard attention mechanisms when combined with Energy-Gated Attention. AI
IMPACT Introduces a new positional encoding method that could improve the performance and efficiency of Transformer models in various NLP tasks.