New research tackles LLM KV cache optimization for efficient inference · 10 sources tracked
ByPulseAugur Editorial·[30 sources]·
Multiple research papers explore novel techniques for optimizing the Key-Value (KV) cache in large language models (LLMs) to improve inference efficiency and reduce memory overhead. These methods include dynamic cache orchestration for AI accelerators, performance characterization of external KV caching with NVMe SSDs, and geometry-aware eviction strategies using complementary diversity metrics. Other approaches focus on fine-tuning models to account for KV cache concatenation, adaptive compression frameworks that adjust to prompt-specific constraints, and methods for reusing KV caches across different contexts and model checkpoints. Some research also investigates attention-weighted value projection and low-rank attention adaptation for quantized KV caches, aiming to recover quality loss while maintaining compression benefits.
AI
IMPACT
These advancements in KV cache management are crucial for enabling more efficient and scalable LLM inference, particularly for long-context workloads.
RANK_REASON
Multiple research papers published on arXiv detailing novel methods for KV cache optimization in LLMs.
arXiv:2512.07312v2 Announce Type: replace-cross Abstract: The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad m…
arXiv cs.LG
TIER_1English(EN)·Joseph Kanichai, Tiziano De Matteis, Animesh Trivedi·
arXiv:2609.11744v1 Announce Type: cross Abstract: Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an exte…
arXiv:2604.11501v2 Announce Type: replace-cross Abstract: Rank reduction discards dimensions; quantization keeps them at lower precision. Comparing the two requires a choice of what compression should preserve. For attention values, we study reconstruction of the attention output…
arXiv cs.LG
TIER_1English(EN)·Michael Wang, Keith Li, Roozbeh Bostandoost·
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs am…
arXiv cs.CL
TIER_1English(EN)·Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi·
arXiv:2609.09768v1 Announce Type: cross Abstract: In Retrieval-Augmented Generation (RAG) systems, a large number of retrieved chunks are concatenated to form the input context so that users can receive high-quality responses based on external knowledge. As a result, the input co…
arXiv:2609.06663v1 Announce Type: cross Abstract: Although multimodal Large Language Models (MLLMs) excel in diverse tasks, their scalability remains limited by the memory and computational overhead of KV cache storage. Recent KV cache eviction approaches incorporate a cosine sim…
arXiv:2609.10266v1 Announce Type: new Abstract: LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt. Two growing workloads break this condition: a retrieval-augmented generation server assembles a different set of retrie…
arXiv:2609.04263v1 Announce Type: cross Abstract: Low-bit key--value (KV) caches reduce the memory required for autoregressive decoding, but the resulting quality loss depends on the model and quantizer. We keep the quantizer fixed and distill the floating-cache model's behavior …
arXiv cs.CL
TIER_1English(EN)·Janghyeon Kim, Minsoo Kim, Kyuhong Shim, Jungwook Choi·
arXiv:2609.04971v1 Announce Type: cross Abstract: Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, o…
arXiv:2609.03235v1 Announce Type: new Abstract: Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooki…
arXiv:2609.03949v1 Announce Type: cross Abstract: The problem. A long-lived KV cache must be compressed before the queries that will read it exist; selection by observed attention (H2O, SnapKV) collapses there (0.00-0.33 needle retrieval on a NoPE MLA model), because a token's im…
arXiv:2609.03430v1 Announce Type: new Abstract: Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score ea…
BeaconKV improves memory efficiency for long reasoning traces by using compact beacon queries to predict which past key-value pairs will be revisited, reducing cache size without sacrificing accuracy.
Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different at…
Random eviction of reasoning tokens matches selective KV cache compression because reasoning traces are self-protecting through redundancy, making scoring unnecessary once prompts are preserved.
arXiv:2609.00891v1 Announce Type: new Abstract: Sharing context between LLMs in a multi-model system requires the receiving model to prefill the shared prefix because KV caches are model-specific. Recent closed-form cross-model KV transfer, hereafter Full-Head Mapping, avoids thi…
arXiv:2604.08426v5 Announce Type: replace-cross Abstract: With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a…
arXiv cs.CL
TIER_1English(EN)·Renato Geh, Alex Chen, Daniel Israel, Aditya Grover, Guy Van den Broeck·
arXiv:2608.28293v1 Announce Type: new Abstract: The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible cost to quality. This holds empirically for many existing methods, though most r…
arXiv:2609.11582v1 Announce Type: new Abstract: As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-only LLMs, but its application to Omni-LLMs remains une…
<p>Production LLM applications rarely receive a question nobody has asked before. Support assistants and RAG pipelines field the same intents thousands of times a day, each phrased differently, and most stacks treat every phrasing as a fresh, fully billed request. Redis LangCache…
<p>Agent loops burn tokens and clock cycles on repeated work. The same LLM prompt fires twice, the same tool call fetches identical data, and session state gets reconstructed from scratch on every request. Agent-cache solves this with a three-tier caching architecture backed by V…
<p>A model can advertise a 128K context window and still fail at 40K tokens on a 16 GB GPU. The architecture ceiling never promised that weights, KV cache, compute buffers, and the desktop compositor would fit on your card at the same time.</p> <p>The KV cache is usually where lo…
<h2> Title: Performance Loss Analysis of Domestic KV Cache Products in Multi-Data Center Deployments </h2> <h2> Abstract: This article analyzes the performance loss of domestic KV Cache products in multi-data center deployments. Through actual measured data from Xinyuanyi and pub…
<h2> Title: Trends and Market Demand Analysis of Domestic KV Cache </h2> <h2> Introduction </h2> <p>As the demand for computing power continues to grow, domestic KV Cache, as an efficient data storage and access technology, is gradually becoming a focus of market attention. This …
<h2> TITLE: How KV Cache Pooling Sharing Enhances Resource Utilization in Large-scale Model Inference </h2> <h2> DESC: Analyzing the application of KV Cache pooling sharing in large-scale model inference and discussing its impact on resource utilization. </h2> <h2> SLUG: kv-cache…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>The latency and energy bottleneck in small-model inference often lies not in compute but in memory access and VRAM management. A well-designed KV Cache tiering and disaggregated storage architecture can improve both time-to-first-token (TTFT) and energy per unit of throughput.…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>Spatial locality optimization for KV Cache is evolving from in-HBM paging management to a tiered architecture spanning multiple storage layers. Measured on the Mingxin FX100 under a 480B production deployment with long-context cold-recovery workloads, tiered acceleration impro…
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1w8jflp/block_kv_cache_streaming_bound_vram_at_long/"> <img alt="Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena by giveen · Pull Request #357 · TheTom/llama-cpp-turboquant" …
