PulseAugur
中
实时 16:11:17
English(EN) Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

FairInference 系统为多租户 LLM 服务提供新颖的延迟隔离

研究人员推出 FairInference,一个旨在为多租户 LLM 服务提供强大延迟隔离的新颖系统。与专注于吞吐量公平性的现有解决方案不同,FairInference 保证了一个行为良好的客户端的 token 生成时间不会超过其隔离的生成时间超过指定的 delta ($\delta$)。这是通过强制执行每个 token 的截止日期以及管理来自共享 GPU 资源和 KV 缓存的延迟来实现的。该系统旨在防止一个客户端的高需求工作负载降低其他客户端的性能,同时提高整体吞吐量。 AI

影响 这项研究可能为共享 LLM 服务的用户带来更可靠、可预测的性能,改善用户体验,并可能支持依赖严格延迟保证的新应用。

排序理由 该集群描述了 arXiv 论文中提出的一种用于提高 LLM 服务性能的新颖系统。

在 Hugging Face Daily Papers 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →

FairInference 系统为多租户 LLM 服务提供新颖的延迟隔离

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
该集群描述了 arXiv 论文中提出的一种用于提高 LLM 服务性能的新颖系统。
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
infra, paper
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
63 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

报道来源 [2]

  1. arXiv cs.LG TIER_1 English(EN) · Dev Bali, Soujanya Ponnapalli, Yichuan Wang, Natacha Crooks, Scott Shenker, Matei Zaharia ·

    Token 延迟公平性:多租户 LLM 服务的性能隔离

    arXiv:2609.18112v1 Announce Type: cross Abstract: LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput i…

  2. Hugging Face Daily Papers TIER_1 English(EN) ·

    Token 延迟公平性:多租户 LLM 服务的性能隔离

    LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and b…