LLM Serving
PulseAugur coverage of LLM Serving — every cluster mentioning LLM Serving across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New system Gandalf extends SSD lifetime for LLM KV cache offloading
Researchers have developed Gandalf, a novel system designed to extend the lifespan of Solid State Drives (SSDs) used for offloading KV caches in Large Language Model (LLM) serving. Traditional LLM serving methods often …
-
High Bandwidth Flash Boosts LLM Serving Performance, Research Finds
A new research paper explores the use of high-bandwidth flash (HBF) to augment High Bandwidth Memory (HBM) for Large Language Model (LLM) serving. The study introduces a hierarchical storage system and a buffered cache-…
-
FairInference system offers novel latency isolation for multi-tenant LLM serving
Researchers have introduced FairInference, a novel system designed to provide strong latency isolation for multi-tenant LLM serving. Unlike existing solutions that focus on throughput fairness, FairInference guarantees …
-
New LLM routing framework optimizes performance using online linear programming · 2 sources tracked
A new online linear programming framework has been developed for optimizing routing in large language model (LLM) serving. This approach uses a multi-objective optimization strategy to balance latency, throughput, and o…
-
New systems tackle LLM long-context serving bottlenecks
Two new research papers introduce novel approaches to managing the KV cache, a critical bottleneck in serving large language models with long contexts. RedKnot proposes a head-aware KV cache management system that decom…