研究人员推出了一种新颖的混合模型Oryx，该模型能够灵活地在给定序列中不同序列混合器（如二次Attention和线性循环）之间进行切换。这种方法允许通过Attention实现丰富的上下文利用，通过线性循环实现高效生成，同时跨模式共享超过90%的参数。通过Mamba-2和Gated DeltaNet变体（高达1.4B模型）进行的验证表明，Oryx在语言建模任务上取得了与单一混合器基线相当或更优的性能，并在检索任务上以显著更少的Attention模式处理的token匹配了Transformer基线性能。 AI
影响 引入了一种新颖的混合架构，可能提高长上下文序列建模的效率和性能。
排序理由 该集群包含一篇详细介绍新模型架构的研究论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2605.28769v1 Announce Type: new Abstract: Softmax attention is the cornerstone of modern large language models, but its memory scales linearly and compute quadratically with sequence length. Linear recurrent models, such as linear attention and state space models, have beco…
Softmax attention is the cornerstone of modern large language models, but its memory scales linearly and compute quadratically with sequence length. Linear recurrent models, such as linear attention and state space models, have become widely studied as alternatives to attention d…