LLM Serving
PulseAugur coverage of LLM Serving — every cluster mentioning LLM Serving across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
FairInference system offers novel latency isolation for multi-tenant LLM serving
Researchers have introduced FairInference, a novel system designed to provide strong latency isolation for multi-tenant LLM serving. Unlike existing solutions that focus on throughput fairness, FairInference guarantees …
-
New LLM routing framework optimizes performance using online linear programming · 2 sources tracked
A new online linear programming framework has been developed for optimizing routing in large language model (LLM) serving. This approach uses a multi-objective optimization strategy to balance latency, throughput, and o…
-
New systems tackle LLM long-context serving bottlenecks
Two new research papers introduce novel approaches to managing the KV cache, a critical bottleneck in serving large language models with long contexts. RedKnot proposes a head-aware KV cache management system that decom…