研究人员开发了一种新颖的Transformer模型线性化方法，解决了因果自注意力带来的二次成本问题，该问题阻碍了长上下文推理。该方法分离了状态更新设计的关键影响，表明softmax依赖于键依赖的、秩为1的正交投影。通过引入sink tokens、短卷积和固定预算缓存路由等结构干预，该方法显著降低了近似误差。该线性化技术应用于多达32B参数的LLaMA和Qwen模型，在MMLU上的表现优于之前的事后基线，并在长上下文检索中与复杂的自适应缓存框架相媲美。 AI
影响 这项研究可能能够实现Transformer模型中长上下文更有效的处理，从而在需要扩展内存的应用中带来性能的提升。
排序理由 该集群包含一篇详细介绍Transformer模型线性化新方法的学术论文。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2607.07706v1 Announce Type: new Abstract: The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This wo…
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in…
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in…