Recent research explores optimizing Large Language Model (LLM) serving through advanced KV cache management techniques. Papers investigate methods for more efficient KV cache eviction, such as using mid-depth layers of a model as judges for eviction decisions and developing workload-aware managers that adapt eviction policies to different request types. Other research focuses on novel storage architectures, like FlatCake, which proposes a unified address space for DRAM and SSD to overcome the limitations of traditional tiered storage, and ARK, which addresses routing collisions in disaggregated LLM inference by coordinating KV cache transfers.
AI
IMPACT
Optimizations in KV cache management and storage architectures are crucial for reducing LLM serving costs and improving inference speed, enabling wider adoption and more complex applications.
RANK_REASON
The cluster consists of multiple academic papers detailing novel research into LLM serving and KV cache optimization.
arXiv:2611.07542v1 Announce Type: new Abstract: In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the high-bandwidth memory (HBM) capacity wall and results in low conc…
arXiv:2611.05903v1 Announce Type: new Abstract: KV cache eviction is not merely a token-ranking problem: the workspace retained for generating eviction queries also determines the quality of those queries. Consequently, an early workspace error can distort later query observation…
arXiv:2611.06179v1 Announce Type: cross Abstract: AI clusters keep data in remote object stores and rely on local caches to stop GPUs from waiting for data. A cache performs well only when its prefetching and eviction policies match the access pattern, yet these patterns differ a…
arXiv:2611.06397v1 Announce Type: new Abstract: Non-prefix KV-cache reuse encodes retrieved chunks independently and concatenates their caches. To find where a small repair budget should be spent, we vary the chunks around the answer-bearing one (the target) across three 7B-8B mo…
arXiv:2611.01077v1 Announce Type: cross Abstract: Semantic caching lowers the cost of serving large language models (LLMs) by reusing stored responses for similar requests. In multi-turn conversations, whether a cached response is acceptable depends on both conversations' histori…
arXiv:2611.02230v1 Announce Type: cross Abstract: Key-value (KV) cache eviction methods rank tokens by attention scores, and each layer scores its own cache. To our knowledge, no eviction method asks which layer should score. We find, across twenty-one models, that the depth of t…
KV cache eviction is not merely a token-ranking problem: the workspace retained for generating eviction queries also determines the quality of those queries. Consequently, an early workspace error can distort later query observations and reinforce an incorrect final eviction deci…
arXiv cs.CL
TIER_1English(EN)·Jonas Loos, Oliver Eberle·
arXiv:2610.41480v1 Announce Type: new Abstract: Fine-tuning a language model changes how it responds, but where is this fine-tuned behavior instantiated? To better understand this, we evaluate whether the change in model behavior arises only during the generation of the answer to…
arXiv cs.AI
TIER_1English(EN)·Hung-Chun Lin, Ting-Wei Hsu, Chung-En Ho, Ahmed Saeed·
arXiv:2610.41160v1 Announce Type: cross Abstract: Disaggregated LLM inference separates the prefill and decode phases across GPU pools, generating massive KV-cache transfers. Because these transfers last hundreds of milliseconds to seconds, they behave as mega elephant flows that…
Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache evicti…
<p>Every LLM you've used, ChatGPT, Claude, Gemini, Llama, does the same thing under the hood. It predicts one token, appends it to the text, and runs again. It's your phone keyboard's next-word suggestion, in a loop, a few hundred times per answer.</p> <p>Done naively, that loop …
dev.to — LLM tag
TIER_1English(EN)·Gurdeep Dhillon·
<p>Large language models are expensive to run and slow to respond compared with almost any other component in a modern application stack. A single completion can cost orders of magnitude more than a database query and take a thousand times longer. As soon as an application serves…
<p>Anthropic's Batch API halves the price of Claude Sonnet 5.5 if you can wait up to 24 hours. Prompt caching cuts the input side by 95% on every hit, in real time. On a typical month, the two end up close to the same bill, so it's worth knowing which lever you're actually pullin…
dev.to — LLM tag
TIER_1English(EN)·Felipe 0liveira·
<p>This digest covers LLM inference and serving developments from <strong>2026-10-28 to 2026-11-04</strong>. The pattern this week is hard to miss: almost every serious throughput or cost win came from rethinking how the KV cache and expert weights are managed in memory, not from…
dev.to — LLM tag
TIER_1English(EN)·Sachin Kr. Rajput·
<p>In every part of this series, one thing kept showing up: the <strong>notebook</strong>. In <a href="https://blog.sachinrajput.dev/posts/one-request-end-to-end/" rel="noopener noreferrer">Part 1</a> it grew by 72 KB with every word. In <a href="https://blog.sachinrajput.dev/pos…
dev.to — LLM tag
TIER_1English(EN)·Prabhakar Chaudhary·
<h1> FlatCake: Why Tiered KV Cache Storage Is Holding LLM Serving Back </h1> <p>When you send a long prompt to an LLM API, the service has to store the key-value (KV) cache for your session somewhere. For short conversations, GPU HBM handles it fine. But as context windows stretc…
