一篇新研究论文介绍了一种名为HCRMap的框架，该框架旨在优化3.5D片上系统上混合专家（MoE）大语言模型的推理。HCRMap通过动态管理不同内存层级的专家副本，解决了专家热度倾斜问题（即某些专家接收到的token数量不成比例）。这种方法旨在缓解通信、内存带宽和执行队列中的瓶颈。实验表明，与Hydra++、MoEntwine和PIMoE等现有方法相比，HCRMap显著降低了端到端延迟。 AI
影响 这项研究可能有助于在专用硬件架构上实现更高效、更快速的大语言模型推理。
排序理由 该集群包含一篇详细介绍用于优化AI模型推理的新框架的研究论文。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2607.11586v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens…
Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded…
Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded…