English(EN) The Residual Stream Has a Geometry of Time

Transformer残差流展现时间几何性，集中上下文

作者 PulseAugur 编辑部 · [1 个来源] · 2026-06-06 20:52

研究人员发现，Transformer中的残差流（常被比作工作记忆）展现出与时间相关的独特几何性。通过分析Gemma-2-2B模型，他们发现跨越许多token的持久信息集中在一个低维子空间中，而不是弥散的。这些持久信息对序列顺序高度敏感，因为打乱token会急剧降低这些慢方向的时间尺度。 AI

影响揭示了Transformer可能如何编码时间信息，可能指导未来的模型架构和可解释性方法。

排序理由该集群包含一篇详细介绍Transformer模型内部实验发现的研究论文。[lever_c_demoted from research: ic=1 ai=1.0]

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 0 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇详细介绍Transformer模型内部实验发现的研究论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, other; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: 81 days old; Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

LessWrong (AI tag) TIER_1 English(EN) · Fodenthal · 2026-06-06 20:52

残差流具有时间几何

<h1>Preface</h1> <p>This is a preliminary writeup for an experiment on residual stream geometry. The research direction seems pretty underexplored, so I’m posting early to collect objections, research intuitions, and connections to problems other people are thinking about before …