研究人员提出了一种新颖的Transformer架构，称为“> <former”或“x形”架构，它偏离了所有层之间标准的统一宽度。这种新设计为早期和晚期层分配了更宽的容量，同时缩小了中间层，并使用无参数的残差重缩放机制。实证结果表明，这种不均匀的宽度分配在语言模型中带来了更高的性能和更大的资源效率，并减少了FLOPs和KV缓存内存。 AI
影响 通过优化参数和计算分配，这种架构可能带来更具资源效率的大型语言模型。
排序理由 该集群描述了一篇在arXiv上发表的关于新颖Transformer架构的研究论文。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
arXiv:2606.18246v1 Announce Type: new Abstract: Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across all layers, allocating a fixed parameter and comput…
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across all layers, allocating a fixed parameter and computation budget evenly despite different layers pot…
A novel transformer architecture with nonuniform width allocation across layers achieves better performance and efficiency compared to uniform designs by utilizing a parameter-free residual resizing mechanism.