研究人员提出了一种新颖的、类似“睡眠”的巩固机制，用于基于 Transformer 的大型语言模型，以解决注意力机制在上下文长度上扩展性差的问题。该方法包括定期将近期上下文转换为持久的快速权重并清除键值缓存。在“睡眠”期间，模型执行离线循环传递来更新状态空间模型块，将计算转移到此阶段，同时保持推理速度。该方法在需要更深层推理的任务上显示出改进的性能，特别是随着睡眠时间的增加。 AI
影响 这项研究通过在不牺牲推理速度的情况下改进上下文处理，有望实现更高效、更强大的 LLM 来执行长时任务。
排序理由 该集群包含一篇详细介绍 LLM 新机制的研究论文。
在 Hugging Face Daily Papers 阅读 →
AI 生成摘要 · Google Gemini · 来自 5 个来源。 我们如何撰写摘要 →
arXiv:2605.26099v1 Announce Type: cross Abstract: Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a m…
Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into per…
Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into per…
A sleep-like consolidation mechanism for transformer models uses fast weights and recurrent passes to improve long-context processing while maintaining inference speed.
Language Models Need Sleep https://arxiv.org/abs/2605.26099 # HackerNews # Tech # AI