PulseAugur
中
实时 20:18:09
English(EN) InferScale: GPU-Native KV Injection for Personalized LLM Serving

InferScale系统通过可重用的KV状态优化LLM服务

作者 PulseAugur 编辑部 · [2 个来源] ·

研究人员开发了InferScale，一个新颖的GPU原生系统，旨在增强个性化大型语言模型（LLM）的服务。InferScale通过利用可重用的KV状态来解决内存系统中重复的提示预填充问题，从而在检索到的内存大小增加时减少首次令牌生成时间（TTFT）。该系统与vLLM集成，无需修改引擎或对模型进行微调，并在各种开源模型上展示了TTFT和吞吐量的显著改进。 AI

影响 降低了个性化LLM的服务延迟并提高了吞吐量，可能支持更具响应性和可扩展性的AI应用。

排序理由 该集群描述了一种在arXiv论文中详细介绍的、用于提高LLM服务效率的新技术方法。

在 Hugging Face Daily Papers 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →

InferScale系统通过可重用的KV状态优化LLM服务

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
该集群描述了一种在arXiv论文中详细介绍的、用于提高LLM服务效率的新技术方法。
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
infra, paper
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
86 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

报道来源 [2]

  1. arXiv cs.LG TIER_1 English(EN) · Peter Li, Prashant Pandey ·

    InferScale：面向个性化 LLM 服务的 GPU 原生 KV 注入

    arXiv:2607.27090v1 Announce Type: cross Abstract: Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g…

  2. Hugging Face Daily Papers TIER_1 English(EN) ·

    InferScale：面向个性化LLM服务的GPU原生KV注入

    Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subs…

相关实体

相关话题