MeshKV architecture boosts transformer decoding with novel NoC KV cache fabric

Researchers have developed MeshKV, a novel network-on-chip (NoC) architecture designed to accelerate transformer decoding by optimizing the movement of key-value (KV) caches. This system addresses bottlenecks in traditional tiled accelerators by treating KV cache blocks as packetized flows, reducing traffic and improving bandwidth utilization. MeshKV has demonstrated significant performance gains, including up to 58% reduction in interconnect traffic and a 1.9x increase in multi-stream throughput on FPGA implementations using models like LLaMA-2 7B and Mistral-7B. AI

影响 Optimizes KV cache movement for transformer models, potentially improving inference speed and scalability for large language models.

排序理由 The cluster describes a novel architecture presented in a research paper on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

• arXiv
• field-programmable gate array
• Hugging Face
• LLaMA-2 7B
• MeshKV
• mistral:7b
• transformer

• 基础设施
• 论文

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →