New research tackles LLM KV cache compression for efficient long-context inference · 5 sources tracked
ByPulseAugur Editorial·[15 sources]·
Multiple research papers released on arXiv propose novel methods to compress Key-Value (KV) caches for large language models, addressing memory bottlenecks during long-context inference. Techniques like BeaconKV, SGD-KV, VestigeKV, and Random Attention aim to reduce memory usage and improve throughput by intelligently selecting or compressing KV cache data. These approaches vary from using beacon queries and summarization-guided diagnostics to random eviction and leveraging vestigial signals, all seeking to maintain accuracy while significantly cutting down memory requirements.
AI
IMPACT
These advancements in KV cache compression could significantly improve the efficiency and scalability of large language models, enabling more complex reasoning and longer context windows.
RANK_REASON
Multiple research papers published on arXiv introducing new methods for KV cache compression in LLMs.
arXiv:2609.04971v1 Announce Type: cross Abstract: Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, o…
arXiv:2609.04263v1 Announce Type: cross Abstract: Low-bit key--value (KV) caches reduce the memory required for autoregressive decoding, but the resulting quality loss depends on the model and quantizer. We keep the quantizer fixed and distill the floating-cache model's behavior …
arXiv:2609.03949v1 Announce Type: cross Abstract: The problem. A long-lived KV cache must be compressed before the queries that will read it exist; selection by observed attention (H2O, SnapKV) collapses there (0.00-0.33 needle retrieval on a NoPE MLA model), because a token's im…
arXiv:2609.03235v1 Announce Type: new Abstract: Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooki…
arXiv:2609.03430v1 Announce Type: new Abstract: Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score ea…
BeaconKV improves memory efficiency for long reasoning traces by using compact beacon queries to predict which past key-value pairs will be revisited, reducing cache size without sacrificing accuracy.
Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different at…
Random eviction of reasoning tokens matches selective KV cache compression because reasoning traces are self-protecting through redundancy, making scoring unnecessary once prompts are preserved.
arXiv:2609.00891v1 Announce Type: new Abstract: Sharing context between LLMs in a multi-model system requires the receiving model to prefill the shared prefix because KV caches are model-specific. Recent closed-form cross-model KV transfer, hereafter Full-Head Mapping, avoids thi…
arXiv:2604.08426v5 Announce Type: replace-cross Abstract: With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a…
arXiv cs.CL
TIER_1English(EN)·Renato Geh, Alex Chen, Daniel Israel, Aditya Grover, Guy Van den Broeck·
arXiv:2608.28293v1 Announce Type: new Abstract: The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible cost to quality. This holds empirically for many existing methods, though most r…
<p>The latency and energy bottleneck in small-model inference often lies not in compute but in memory access and VRAM management. A well-designed KV Cache tiering and disaggregated storage architecture can improve both time-to-first-token (TTFT) and energy per unit of throughput.…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>Spatial locality optimization for KV Cache is evolving from in-HBM paging management to a tiered architecture spanning multiple storage layers. Measured on the Mingxin FX100 under a 480B production deployment with long-context cold-recovery workloads, tiered acceleration impro…
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1w8jflp/block_kv_cache_streaming_bound_vram_at_long/"> <img alt="Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena by giveen · Pull Request #357 · TheTom/llama-cpp-turboquant" …
