SemiAnalysis 正在探索语言模型的先进位置嵌入技术，超越像 RoPE（旋转位置嵌入）这样的标准方法。该分析深入研究了包括群论在内的数学框架，以理解和潜在地改进模型处理序列数据的方式。虽然 RoPE 被 DeepSeek 和 Qwen 等模型广泛使用，但研究表明，替代的、更复杂的嵌入可能会带来好处，尽管它们伴随着计算上的权衡。 AI
影响 探讨了位置嵌入的新颖数学方法，可能改进未来语言模型的序列处理能力。
排序理由 该集群讨论了用于语言模型的关于位置嵌入的数学研究，并引用了学术论文和概念。
AI 生成摘要 · Google Gemini · 来自 7 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
Resources mentioned: 🟠 Using group theory to explore the space of positional encodings for attention (blog): https://t.co/ciWX1bLAWu 🟠 Positional Encodings and Group Theory (video): https://t.co/16BvKWPBU2 🟠 RoFormer: https://t.co/ZXMDVyvCf4 (7/7)
If we really wanted chaos, we could learn independent embedding functions for s and t. We might call this "position invariant," and it starts to cut against the purpose of a positional embedding in the first place. It does not respect translation invariance, but it could be https…
The weakest possible bilinear embedding that respects translation invariance is just a lookup table from the displacement to an arbitrary learnable square matrix of the proper dimension. One reason algebraic structure is often nice is that it saves computation by reducing https:…
What if we relax these assumptions even further? Let's stop requiring separable action. This leaves q(s)T A(t-s) k(t), where A is an arbitrary translation-invariant function that outputs a square matrix. This means we no longer require any group structure in the output space. To…
One function described by this math is called RoPE. It's ubiquitous in modern ML, used in models like DeepSeek and Qwen. We built a tiny Transformer that learned a Shakespeare dataset more quickly when we gave it positional information encoded via RoPE. (3/7) https://t.co/fiG0srS…
Starting with the hard part, let's write down Jane Street's math. Alok Puranik explores positional embeddings that allow the attention score to be written q(s)T F(s)T G(t) k(t), where q and k are the position-free queries and keys, and F and G do the embedding work. This encodes…
Attention has no innate notion of time. Positional embeddings are how language models tell how far apart tokens are. A recent YouTube video by Jane Street and 3Blue1Brown explores the space of embeddings that satisfy common-sense mathematical restrictions. This space is not only…