English(EN) Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

FairInference 系统为 LLM 服务提供严格的延迟隔离

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-17 04:00

研究人员推出了一种名为 FairInference 的新颖系统，旨在为多租户 LLM 服务提供强大的延迟隔离。与以往侧重于长期吞吐量公平性的解决方案不同，FairInference 保证了一个行为良好的客户端的 token 生成时间不会超过其隔离生成时间的指定增量（$\delta$）。这是通过强制执行每个 token 的截止日期并管理来自共享 GPU 资源和 KV 缓存的延迟来实现的，最终在限制延迟峰值的同时提高整体吞吐量。 AI

影响增强了共享 LLM 推理服务的可靠性和可预测性，这对于实时应用程序至关重要。

排序理由该集群包含一篇关于 LLM 服务新系统的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 13 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇关于 LLM 服务新系统的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: infra, paper; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.LG TIER_1 English(EN) · Dev Bali, Soujanya Ponnapalli, Yichuan Wang, Natacha Crooks, Scott Shenker, Matei Zaharia · 2026-09-17 04:00

Token 延迟公平性：多租户 LLM 服务的性能隔离

arXiv:2609.18112v1 Announce Type: cross Abstract: LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput i…

本文如何被排名

报道来源 [1]

Token 延迟公平性：多租户 LLM 服务的性能隔离

相关实体

相关话题