New methods target LLM KV cache compression for efficiency
ByPulseAugur Editorial·[10 sources]·
Researchers are developing advanced techniques to compress the Key-Value (KV) cache in Large Language Models (LLMs), a major contributor to memory costs during inference. New methods like JoLT and FlashJoLT utilize tensor decomposition and residual allocation to achieve significant compression with minimal loss in performance, as demonstrated on models like Mistral-7B-v0.3 and Llama 2 13B. Other research explores how query visibility affects compression method rankings, with some methods performing poorly when queries are not known beforehand. Additionally, methods like PM-KVQ and REAL are designed to handle long context windows and diverse attention behaviors, aiming to reduce cumulative errors and improve efficiency for long-CoT LLMs.
AI
IMPACT
Advances in KV cache compression could significantly reduce inference costs and memory requirements, enabling more efficient deployment of LLMs, especially for long-context tasks.
RANK_REASON
Multiple research papers detailing novel methods for compressing LLM KV caches.
arXiv:2505.18231v3 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache. Vector Quantization (VQ) is recently adopt…
arXiv:2607.11942v1 Announce Type: cross Abstract: KV-cache compression methods are predominantly evaluated with the query appended to the context before compression -- a query-aware protocol. Yet the economic case for a compressed KV cache is reuse: compress a document once, answ…
arXiv:2607.12550v1 Announce Type: cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference. It grows with batch size, context length, and depth, and at long context it, rather than the model weights, sets the ceiling on throughput. Two…
The key-value (KV) cache has become the dominant memory cost of transformer inference. It grows with batch size, context length, and depth, and at long context it, rather than the model weights, sets the ceiling on throughput. Two families of methods reduce it. Low-rank methods f…
arXiv:2607.09683v1 Announce Type: cross Abstract: This study systematically compares Turbo-Quant and SpectralQuant KV-cache compression, evaluating non-dominated schemes, including WHT rotation with Beta Lloyd-Max and QJL, through a statistical validation methodology that separat…
arXiv:2505.18610v2 Announce Type: replace Abstract: Recently, significant progress has been made in developing reasoning-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) techniques. However, this long-CoT reasoning process imposes substantial memory overhe…
arXiv cs.AI
TIER_1English(EN)·Mengjie Li, Yuan Feng, Xike Xie, William J. Song·
arXiv:2508.15806v2 Announce Type: replace-cross Abstract: The growing sequence length of large language models poses significant challenges for key-value (KV) caches. Existing state-of-the-art cache eviction methods primarily analyze the inference behavior of attention heads in s…
<h4>An outlook on how input-dependent step-size updates will shape the next generation of stream-processing AI.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gaLdwPNUza_RP-Gn" /></figure><p><em>Cover infographic visualizing the evolutionary leap from hea…
Medium — MLOps tag
TIER_1English(EN)·Jagadish Mukku·
<p>Most developers know KV cache reduces costs. Few have actually modeled how much, at current pricing, cache hit rate moves the needle on their bill.</p> <p>Take Qwen3.7 Plus on RouteAI: standard input at <strong>$0.24/M tokens</strong>, cache read at <strong>$0.048/M</strong> —…
