arXiv:2608.14191v1 Announce Type: cross Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-preci…
arXiv:2608.13043v1 Announce Type: new Abstract: Diffusion models have achieved dominant performance in visual generation but suffer from substantial inference overhead. While cache-based acceleration has emerged as a promising solution, existing policies rely on local similarity …
arXiv:2608.13263v1 Announce Type: new Abstract: Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction al…
arXiv:2608.11231v1 Announce Type: new Abstract: LLM serving is increasingly accelerated by position-independent caching (PIC). Existing PIC methods, however, are built for full-attention models, where a token-indexed KV cache underlies its core operations: matching reusable token…
arXiv:2608.09412v1 Announce Type: new Abstract: KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why. We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions. First, a 25-metho…
arXiv:2608.07915v1 Announce Type: new Abstract: Large language models (LLMs) increasingly read long inputs in the agentic era, from whole documents and codebases to conversations across many turns. Their inference memory is then dominated by the key-value (KV) cache, the stored a…
arXiv:2608.07855v1 Announce Type: new Abstract: Multi-turn Reasoning-and-Acting (ReAct) agents accumulate growing trajectories of reasoning, tool calls, and observations. Their key-value (KV) caches grow accordingly, increasing memory use and attention cost during model inference…
arXiv:2607.28069v2 Announce Type: replace Abstract: Long-context retrieval and agentic workloads repeatedly reuse the same documents under changing instructions, histories, and document orders. Prefix caching cannot exploit this reuse, while position-independent caching (PIC) rem…
arXiv:2608.08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging. Existing methods rely on proxies such as layer depth, attention statistics, or representatio…
arXiv:2608.07001v1 Announce Type: new Abstract: As large language models (LLMs) process increasingly long contexts, KV cache storage and repeated access have become a major bottleneck. Existing KV cache compression methods rely on predefined, fixed compression rules and are typic…
arXiv cs.AI
TIER_1English(EN)·Gyuwan Kim, Cheoneum Park, Tao Yang·
arXiv:2608.07458v1 Announce Type: cross Abstract: Recent optimization studies on Retrieval-Augmented Generation (RAG) have exploited chunk-level KV cache reuse to avoid processing long retrieved contexts for higher efficiency, while significant information redundancy and noise st…
Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation…
Recent optimization studies on Retrieval-Augmented Generation (RAG) have exploited chunk-level KV cache reuse to avoid processing long retrieved contexts for higher efficiency, while significant information redundancy and noise still remain in the coarse-grained chunks. This pape…
arXiv:2608.04074v1 Announce Type: cross Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and s…
arXiv cs.AI
TIER_1English(EN)·Wonpyo Park, Seung-won Hwang·
arXiv:2608.03276v1 Announce Type: new Abstract: Long-context inference with large language models is constrained by the linear growth of the key-value cache to sequence length. While pruning offers mitigation, prevailing methods determine query-specific token importance that cann…
arXiv:2608.02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important. However, existing rotation-based INT2 methods op…
arXiv:2608.03893v1 Announce Type: new Abstract: Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-…
arXiv cs.LG
TIER_1English(EN)·Jihao Xin, Tian Lyu, David Keyes, Hatem Ltaief, Marco Canini·
arXiv:2602.02599v4 Announce Type: replace Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache. Structured pruning is a direct way to shrink it: dropping the least useful channels of the W_k, W_v pr…
arXiv cs.CL
TIER_1English(EN)·Jialong Han, You Wu, Kewei Tu·
arXiv:2608.00528v1 Announce Type: new Abstract: The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache. Although low-rank compression of KV cac…
arXiv:2608.01247v1 Announce Type: new Abstract: Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are…
arXiv cs.LG
TIER_1English(EN)·Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee, Ion Stoica, Jae W. Lee·
arXiv:2606.21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle. Offloading the cache to CPU DRAM restores capacity, but the limited PCIe bandwidth forces state-of-the-art …
Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained. We introduce RestoreKV, which complem…
Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained. We introduce RestoreKV, which complem…
arXiv cs.LG
TIER_1English(EN)·Stephen Gould, Anton van den Hengel·
arXiv:2607.27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years. Computational demands of large language models (LLMs) and their multi-modal variants during …
arXiv:2607.28495v1 Announce Type: cross Abstract: Stage-replay diagnostics reconstruct intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix. We audit that assumption at a whole reasoning-stage b…
arXiv cs.LG
TIER_1English(EN)·Tan T. Nguyen, Quan V. Dang·
arXiv:2607.24331v1 Announce Type: new Abstract: As the inference phase of Large Language Models (LLMs) requires handling long context windows, the Key-Value (KV) cache initially appears to address this challenge but eventually becomes a significant bottleneck as the context windo…
arXiv cs.LG
TIER_1English(EN)·Chao Fang, Jun Yin, Man Shi, Marian Verhelst·
arXiv:2607.22389v1 Announce Type: cross Abstract: With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this challenge, we propose HiKV, a novel algorithm-hardwa…
arXiv:2607.20507v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost. We present MiniCache, a reusable program…
arXiv cs.AI
TIER_1English(EN)·Pragaash Ponnusamy, Shivam Sahni, Jue Wang, Tri Dao·
arXiv:2607.20475v1 Announce Type: new Abstract: Sampling in LLM inference comprises a combinatorial set of logit processing, token selection, and verification operations for speculative decoding. However, existing implementations either accelerate only subsets of this pipeline, r…
arXiv cs.CL
TIER_1English(EN)·Chuheng Du, Junyi Chen, Hanlin Tang, Kan Liu, Tao Lan, Lin Qu, Chaoyue Niu, Shengzhong Liu, Guihai Chen, Fan Wu·
arXiv:2607.17715v1 Announce Type: new Abstract: Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inference cost, recent work has explored key-value (KV)…
arXiv:2607.15516v1 Announce Type: cross Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent). The compression literature has standardized on query-aware…
arXiv cs.CL
TIER_1English(EN)·Shahrzad Esmat, Dhawal Shah, Ali Jannesari·
arXiv:2607.15498v1 Announce Type: new Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference. Two leading training-free families are both structurally limited: token-selection methods (SnapKV, Ada-KV) score importance…
<p>I recently spent a frustrating afternoon debugging a support bot built on top of LangGraph, where the bot would occasionally take an inordinate amount of time to respond to user queries. The bot's workflow involved multiple tool calls to external services, such as entity disam…
<h4>It was never <em>only</em> about tokens. It’s all about the cache.</h4><blockquote>Read the article for free <a href="https://medium.com/@jordancarson/af9e6b92139e?source=friends_link&sk=d1099fdd2f99ae869b92f3aefce68aab">here</a>.</blockquote><figure><img alt="" src="http…
<h4>The next generation of LLM systems will be shaped not only by their models, but by how well they manage context.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDZLP4ysvvqcmhlnz6alpg.png" /></figure><p>Imagine a production AI assistant that retrieves …
<p>Hi, I'm <a href="https://www.linkedin.com/in/kenichi-matsui-86b5392b/" rel="noopener noreferrer">Matsuken</a>, a data scientist at a Japanese technology company.</p> <p>In this article, I'll explain how decoding works in a Transformer that uses a key-value (KV) cache.</p> <blo…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>The performance gap in KV Cache handling between domestic AI inference accelerators and international counterparts cannot be captured by a single metric. However, a comparable framework can be established through measured data under unified testing conditions. Mingxin FX100, u…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>Different brands of KV Cache storage products in vLLM inference—let's state the conclusion upfront: <strong>without a unified testing methodology, any cross-brand comparison figures lack decision-making value</strong>. The key to comparison is not "who is faster" but whether t…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<h2> Key Findings </h2> <p>The relationship between KV Cache memory investment and inference acceleration gains is not linear: once cache capacity reaches a certain threshold, further expansion yields noticeably narrower improvements in hit rate and end-to-end speedup. This pheno…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<h2> Key Takeaways </h2> <p>In local LLM deployment, there is no universally optimal solution for whether KV Cache stays in VRAM or is offloaded to memory. The boundary is determined by three variables: concurrency level, context length, and the SLA requirement for time-to-first-…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>Domestic KV Cache products are moving from proof-of-concept to large-scale deployment. This article analyzes their application effectiveness, applicable boundaries, and selection criteria in large-model inference, based on measured data from Mingxin's FX100 series in 480B-clas…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>The deployment location and access path of KV Cache are becoming a non-negligible variable in data center energy bills. Based on measured data from Mingxin FX100 under long-context workloads with a 480B model (measured, reports R2/R3), this article presents a methodological fr…
<h2> TL;DR </h2> <p>KV cache eviction permanently throws away tokens. KV cache merging corrupts the values of surviving tokens. ResKV (arXiv:2607.29591) does neither — it splits a fixed budget into a main cache that keeps exact tokens and a residual cache that reconstructs the at…
dev.to — LLM tag
TIER_1English(EN)·Prabhakar Chaudhary·
<h1> SparseSpec-L: How a Sparse KV Cache Makes Long-Context LLM Inference 2.79× Faster — Without Any Training </h1> <p>Speculative decoding has become one of the more practical tools for cutting LLM inference latency. The idea is simple: use a fast draft mechanism to propose seve…
<p>Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value (KV) cache, and if a later request shows up with the same prompt prefix, they reuse that cache inste…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>KV Cache pooling and sharing centralizes fragmented GPU memory resources and allocates them on demand, significantly improving resource utilization for large-model inference. Measured on a 480B-parameter model, Mingxin Technology reports that tiered KV Cache acceleration impro…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>Inference workloads on online education platforms are characterized by long contexts, high concurrency, and strong interactivity. The hit rate and reuse strategy of the KV Cache directly determine service cost and user experience. Measured data from the Mingxin FX100 on a 480B…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>Load balancing for distributed KV Cache is fundamentally a trade-off among <strong>memory capacity, memory access bandwidth, and cross-node communication overhead</strong>. No single strategy is optimal under all workloads: for long-context cold-restart scenarios with a 480B-c…
<h2> 600 MiB of headroom </h2> <p>My RTX 4070 was running Qwen 35B beautifully after the <code>--cpu-moe</code> trick from a previous run. The tokens/sec were where I wanted them. VRAM sat at 11,714 MiB out of 12,281 — 95% full.</p> <p>That leaves 600 MiB. Not enough for a seriou…
