研究人员为大型语言模型（LLMs）引入了一种新颖的双路径架构，旨在有效扩展计算和容量。该架构利用了多次应用的深度子层（具有共享参数）和具有扩展前馈网络的宽子层。每令牌门（Per-token gates）动态路由信息，允许对令牌分配进行详细分析。与同等FLOPs匹配的模型相比，所提出的模型在语言建模和下游任务上表现出更优越的性能，同时使用的参数也更少。 AI
影响 引入了一种新颖的架构，可更有效地扩展LLM的计算和容量，有望实现参数更少但性能更强的模型。
排序理由 该集群包含一篇描述LLM新架构的学术论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2605.30202v1 Announce Type: new Abstract: Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transfo…
Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We propose a novel dual-path block that ca…