PulseAugur
中
实时 07:49:54
English(EN) Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference

LLM路由方法提高效率并降低延迟 · 已追踪2个来源

一篇新研究论文介绍了一种名为HeRo(History-Aware Routing)的动态路由框架,用于大型语言模型。该框架使用一种记忆机制来跨模型深度维护路由状态。这种方法将先前的路由分数聚合为紧凑的历史表示,在Llama 3.1-8B、Llama 2-7B和Llama 2-13B的基准测试中持续优于现有方法。与此同时,Amazon SageMaker Inference推出了前缀感知路由,这是一种将具有相似前缀的请求导向同一实例的策略,显著提高了KV缓存命中率并降低了Llama 3.1 70B等模型的延迟。 AI

影响 路由和缓存策略的这些进步有望显著降低LLM推理成本和延迟,从而实现更高效的部署和更广泛的大型语言模型可访问性。

排序理由 该集群包含一篇关于LLM推理优化的新研究论文以及来自云提供商的相关产品公告。

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →

LLM路由方法提高效率并降低延迟 · 已追踪2个来源

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
该集群包含一篇关于LLM推理优化的新研究论文以及来自云提供商的相关产品公告。
Source corroboration
3 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
infra, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
73 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

完整方法见我们的编辑标准。

报道来源 [3]

  1. arXiv cs.AI TIER_1 English(EN) · Hongjin Lin, Wentao Wan, Keze Wang ·

    动态路由器需要记忆吗?HeRo:用于高效LLM推理的历史感知路由

    arXiv:2609.08189v1 Announce Type: new Abstract: Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, however, treat each routing decision as a local operation conditioned solely on the…

  2. AWS Machine Learning Blog TIER_1 English(EN) · Kareem Syed-Mohammed ·

    使用 Amazon SageMaker Inference 上的前缀感知路由来降低 LLM 延迟

    Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache h…

  3. Mastodon — mastodon.social TIER_1 English(EN) · aitools2u ·

    🤖 【AWS ML Blog】通过 Amazon SageMaker Inference 上的前缀感知路由降低 LLM 延迟 Amazon SageMaker Inference 现在提供前缀感知路由,一种路由

    🤖 【AWS ML Blog】Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prom... # AI # TechNews 🔗 https:// aws.amazon.com/blogs/machine-l earning/…