PulseAugur
中
实时 13:16:02
English(EN) QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

新方法解决LLM KV缓存压缩问题,实现高效长上下文推理 · 跟踪10个来源

多篇研究论文介绍了压缩大型语言模型(LLM)中KV缓存的新颖技术,KV缓存是高效长上下文推理的关键组成部分。这些方法旨在通过智能管理KV缓存来减少内存使用并提高推理速度。方法包括时间预取、具有专用查询和键变换的低比特量化、跨循环使用残差状态、基于注意力几何的自适应剪枝以及LLM引导的策略演化。其他策略侧重于优化驱逐时机和使用蒙特卡洛估计来预测未来令牌效用。 AI

影响 KV缓存压缩方面的这些进展对于实现更高效、可扩展的大型语言模型部署至关重要,尤其是在需要长上下文理解的任务中。

排序理由 多篇arXiv论文提出了LLM中KV缓存压缩的新方法。

在 Hugging Face Daily Papers 阅读 →

AI 生成摘要 · Google Gemini · 来自 84 个来源。 我们如何撰写摘要 →

新方法解决LLM KV缓存压缩问题,实现高效长上下文推理 · 跟踪10个来源

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
多篇arXiv论文提出了LLM中KV缓存压缩的新方法。
Source corroboration
84 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
infra, paper
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
57 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+36 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

完整方法见我们的编辑标准。

报道来源 [84]

  1. arXiv cs.CL TIER_1 English(EN) · Wanyang Hong, Zhaoning Zhang ·

    压缩不止于选择:稀疏KV缓存的后选择位置几何

    arXiv:2610.21126v1 Announce Type: new Abstract: Sparse KV compression reduces long-context inference costs by retaining selected historical states, yet these states typically keep their coordinates from the full history. Under Rotary Position Embedding (RoPE), gaps left by evicte…

  2. arXiv cs.AI TIER_1 English(EN) · Aozhong Zhang, Selcuk Gurses, Yanxia Deng, Naigang Wang, Chi-Chun Liu, Davis Wertheimer, Derrick Liu, Xin Li, Zi Yang, Felix X. -F. Ye, Penghang Yin ·

    DropKV:解耦残差输出扰动以实现近乎最优的 KV 缓存驱逐

    arXiv:2610.22130v1 Announce Type: new Abstract: Key-value (KV) cache management has become a critical bottleneck for scaling Large Language Models (LLMs) to long-context scenarios due to linear memory growth. While recent eviction methods have moved beyond simple attention-score …

  3. arXiv cs.LG TIER_1 English(EN) · Muhammad Ibrahim ·

    当KV缓存比GPU慢时:为上下文并行预填充扩展远程KV层

    arXiv:2610.21500v1 Announce Type: cross Abstract: Reusing the attention KV cache across requests is a standard way to cut prefill cost in long-context LLM serving, and production systems increasingly back the per-engine cache with a remote tier shared by every engine. We study su…

  4. arXiv cs.LG TIER_1 English(EN) · Adam Deng, Shicheng Liu, Yada Zhu ·

    差不多等于差远了:KV缓存中的保真度-可消耗性解耦

    arXiv:2610.22016v1 Announce Type: new Abstract: Multi-agent LLM systems communicate in text, which is lossy and wasteful: the sender throws away the reading it already did. A growing literature instead transmits the KV cache, the natural carrier of that reading. We study the most…

  5. Hugging Face Daily Papers TIER_1 English(EN) ·

    DropKV:解耦残差输出扰动以实现近乎最优的 KV 缓存驱逐

    Key-value (KV) cache management has become a critical bottleneck for scaling Large Language Models (LLMs) to long-context scenarios due to linear memory growth. While recent eviction methods have moved beyond simple attention-score heuristics to incorporate value representations,…

  6. arXiv cs.AI TIER_1 English(EN) · Han Yu, Wenhui Zhu, Jincheng Cao, Rui Cai, Yuanda Xu, Zhengze Zhou, Xiwen Chen, Hao Wang, Xuanzhao Dong, Han Shi, Menglin Zhou, Shayan Mohajer Hamidi, Lin Su, Mina Razghandi, Sayan Ghosh, Qing Lan, Rohit Jain, Alborz Geramifard ·

    面向不断变化的上下文的预算 KV 缓存修复

    arXiv:2610.17338v1 Announce Type: new Abstract: Language-model agents revise their working context as information becomes obsolete, but deleting text does not erase its influence on cached computation. Later key-value (KV) states remain conditioned on the deleted content, while r…

  7. arXiv cs.AI TIER_1 English(EN) · Yaoyang Liu, Shuwei Shen, Jincheng Tang, Jinguang Liu, Jinbao Xue, Hailong Yang, Ke Zhang, Yinjun Wu ·

    BELT:基于学习到的 Top-k 秩分配的、行为对齐的低秩 KV 缓存压缩

    arXiv:2610.17966v1 Announce Type: cross Abstract: Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather t…

  8. arXiv cs.AI TIER_1 English(EN) · Sasho Nedelkoski, Alexander Acker, Soeren Becker, Dominik Scheinert, Odej Kao ·

    E8 Lattice 关键量化用于低比特 KV 缓存压缩

    arXiv:2610.15384v1 Announce Type: cross Abstract: Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scalar quantization methods achieve significant compression, they typically face a qua…

  9. arXiv cs.AI TIER_1 English(EN) · Sihwan Park, Sung-Yub Kim, Doohyuk Jang, Eunho Yang ·

    KV4-RL:通过 INT4 KV 缓存量化实现 RLVR 更快的异步部署

    arXiv:2610.14448v1 Announce Type: cross Abstract: Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer…

  10. arXiv cs.CL TIER_1 English(EN) · Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Jianchao Tan, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong ·

    OScaR:LLM及更广泛领域的极端KV缓存量化的奥卡姆剃刀

    arXiv:2605.19660v3 Announce Type: replace-cross Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficient deployment. While the established per-ch…

  11. arXiv cs.CL TIER_1 English(EN) · Gil Jae Song, Sungjoo Yoo ·

    PN-Mix:配对范数引导混合精度键缓存量化,用于高效长输出LLM推理

    arXiv:2610.19605v1 Announce Type: cross Abstract: Long-output inference in large language models suffers from severe KV cache memory and bandwidth bottlenecks. While KV cache quantization alleviates this issue, low-bit quantization (e.g., 2-bit) severely degrades model accuracy. …

  12. arXiv cs.AI TIER_1 English(EN) · Xiangyu Li, Huaizhi Tang, Xin Ding, Weijun Wang, Ting Cao, Yunxin Liu ·

    OxyGen:多任务并行下VLA推理的统一KV缓存管理

    arXiv:2603.14371v4 Announce Type: replace-cross Abstract: Embodied AI agents increasingly require parallel execution of multiple tasks, such as manipulation, conversation, and memory construction, from shared observations under distinct time constraints. Recent Mixture-of-Transfo…

  13. arXiv cs.AI TIER_1 English(EN) · Yuechuan Chen, Yanda Guo, Xiaoxi Liu, Weilin Ruan, Huanbo Xu, Ruyi Wang, Wei Zhu ·

    ChampionKV:KV缓存压缩的头部自适应预算分配

    arXiv:2610.18026v1 Announce Type: cross Abstract: Some observation-window-based KV cache compressors use prompt attention to identify important tokens but allocate uniform or predefined budgets across heads, overlooking input-dependent differences in budget demand. We propose Cha…

  14. arXiv cs.AI TIER_1 English(EN) · Zhiwei Chen, Zhongtao Rao, Waikit Xiu, Yuxiang Zhang, Shuting Cui, Jian Chen, Tianxiang Zhao ·

    LatentRemedy:多智能体系统中非前缀 KV 缓存复用的高效方法

    arXiv:2610.18014v1 Announce Type: cross Abstract: Non-prefix KV cache reuse is a promising approach for improving inference efficiency in multi-agent systems as agents share the same context blocks. However, when agents have conflicting tasks, directly reusing non-prefix KV cache…

  15. Hugging Face Daily Papers TIER_1 English(EN) ·

    当 KV 缓存比 GPU 更慢时:为上下文并行预填充扩展远程 KV 层

    Remote KV caching for long-context LLM serving can severely degrade latency unless the store is sharded by parallel rank, loaded asynchronously, and paired with high-bandwidth retrieval, yielding large throughput gains.

  16. Hugging Face Daily Papers TIER_1 English(EN) ·

    PN-Mix:配对归一化引导混合精度键缓存量化,实现高效长输出LLM推理

    PN-Mix improves low-bit KV cache quantization by identifying salient RoPE channel pairs via query-key norms, achieving strong accuracy at lower bit rates.

  17. Hugging Face Daily Papers TIER_1 English(EN) ·

    MultiAlloc:面向KV缓存量化的预算感知精确分配

    The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, includ…

  18. arXiv cs.LG TIER_1 English(EN) · Albert Ge, Akash Haridas, Utkarsh Saxena, Ruijing Yang, Ian Colbert, Frederic Sala, Vikram Appia, Emad Barsoum ·

    CORAL-KV: 紧凑型跨层预测与残差编码用于自回归视频KV缓存

    arXiv:2610.12514v1 Announce Type: cross Abstract: Autoregressive video diffusion models maintain a growing attention key-value (KV) cache, making memory a key bottleneck for long video generation. Recently, clustering-based quantization has emerged as an effective method to compr…

  19. arXiv cs.CL TIER_1 English(EN) · Taiga Ikeda, Daisuke Miyashita, Jun Deguchi ·

    KV-Repurpose:一个用于评估非前缀 KV 缓存重用的测试平台

    arXiv:2610.13060v1 Announce Type: new Abstract: KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We p…

  20. Hugging Face Daily Papers TIER_1 English(EN) ·

    KV4-RL:通过 INT4 KV 缓存量化解锁 RLVR 更快的异步部署

    Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step…

  21. Hugging Face Daily Papers TIER_1 English(EN) ·

    KV-Repurpose:用于评估非前缀 KV 缓存重用的测试平台

    KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We present KV-Repurpose, an open-source, vLLM-based …

  22. arXiv cs.AI TIER_1 English(EN) · Yufeng Wang ·

    冻结解码器,修复编码器:基于SVD的KV缓存压缩的参数高效适配

    arXiv:2610.10552v1 Announce Type: cross Abstract: Comparing parameter-efficient fine-tuning recipes under a single, shared learning rate is a common but flawed practice: when the arms being compared have very different trainable-parameter counts, a shared rate can simultaneously …

  23. arXiv cs.AI TIER_1 English(EN) · Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel ·

    RaReCache:通过基于秩不一致的 선택性再计算弥合跨模型KV缓存重用中的差距

    arXiv:2610.11358v1 Announce Type: new Abstract: Cross-model KV-cache reuse remains a key challenge in modern LLM serving. Coding agents and multi-model systems increasingly route a shared context across models: a user may switch models mid-session, or a cascade may escalate a dif…

  24. arXiv cs.CL TIER_1 English(EN) · Haeyong Kang, Chang D. Yoo ·

    何时驱逐而非保留:用于无训练KV缓存压缩的草稿引导驱逐

    arXiv:2609.33334v2 Announce Type: replace-cross Abstract: Training-free KV-cache compression methods such as SnapKV, H2O, and PyramidKV evict tokens at the end of prefill, aiming to preserve the attention mass that future queries are expected to use -optimizing what to keep. We s…

  25. arXiv cs.CL TIER_1 English(EN) · Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan ·

    阅读要点:查询自适应量化用于 KV 缓存

    arXiv:2610.11245v1 Announce Type: cross Abstract: KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV sto…

  26. arXiv cs.CL TIER_1 English(EN) · Kaicheng Xiao, Liran Dong, Haotian Li, Guoliang Xing ·

    融合KV缓存量化与线性注意力:从理论到预训练权重迁移

    arXiv:2610.11214v1 Announce Type: cross Abstract: KV-cache quantization and linear attention are two representative approaches to tackling the storage and computational costs of Transformers. KV-cache quantization compresses individual KV entries into discrete codes but retains a…

  27. arXiv cs.CL TIER_1 English(EN) · Neha Verma, Sungwon Kim, Kenton Murray, Kevin Duh ·

    VFold:对称感知跨层值缓存压缩

    arXiv:2610.12338v1 Announce Type: new Abstract: While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities.…

  28. Hugging Face Daily Papers TIER_1 English(EN) ·

    VFold:对称感知跨层值缓存压缩

    While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities. However, most existing techniques necessitate a…

  29. arXiv cs.LG TIER_1 English(EN) · Linfeng Dong ·

    KVFetch:KV缓存压缩的缺失一半的时间预取

    arXiv:2610.08811v1 Announce Type: new Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and o…

  30. arXiv cs.LG TIER_1 English(EN) · Davis Wertheimer, Haochen Shen, Ahan Gupta, Derrick Liu, Yu Chin Fabian Lim, Mudhakar Srivatsa, Raghu K. Ganti, Minjia Zhang, Naigang Wang ·

    一种自剪枝Transformer:具有通用注意力的极端KV缓存压缩

    arXiv:2610.09051v1 Announce Type: new Abstract: The large KV-cache size of modern LLMs creates a barrier to efficient deployment. Recent work has explored replacing attention layers' RoPE positional embeddings with alternative decay-based mechanisms, which can then be used to pru…

  31. arXiv cs.LG TIER_1 English(EN) · Sunjoo Whang, Jungjun Oh, Minsung Kim, Dongho Seo, Jisu Shin, Gregory Kielian, Hoi-Jun Yoo, Sangjin Kim ·

    Dual-QK:用于可剪枝2位KV缓存的锐利查询和扁平键

    arXiv:2610.09827v1 Announce Type: new Abstract: Long inputs and extended generation increase the storage and access costs of the key-value (KV) cache. Low-bit quantization reduces storage and memory traffic, while query-channel pruning can further reduce key-cache reads. Rotation…

  32. arXiv cs.LG TIER_1 English(EN) · Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han, Insu Han, Sehoon Kim ·

    ResidualQuant:用于具有 2 位残差的循环 Transformer 的 KV 缓存量化

    arXiv:2610.10381v1 Announce Type: new Abstract: Looped Transformers improve parameter efficiency by repeatedly applying shared Transformer blocks over multiple recurrent loops, increasing computational depth without increasing the parameter count. However, KV cache memory still s…

  33. arXiv cs.AI TIER_1 English(EN) · Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer, Wajih Hassan Raza, Atta Ul Asad, Young D. Kwon, Michal Valko, Dean F. Hougen ·

    KV 缓存驱逐的蒙特卡洛估计

    arXiv:2610.07643v1 Announce Type: cross Abstract: Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter while answering? Since decoding queries are unavailable at eviction time, prior futu…

  34. arXiv cs.AI TIER_1 English(EN) · Kaushal Mhapsekar, Bita Aslrousta, Brijesh Kumar Bhayana, Paula Contreras, Azam Ghanbari, Ethan Goodman, Anna Andriiko, Samira Mirbagher Ajorpaz ·

    CACHEFORGE:LLM驱动的端到端生成式缓存替换策略,以实现高性能和硬件效率

    arXiv:2610.07668v1 Announce Type: cross Abstract: Modern cache replacement designs saturate because they operate within fixed representational structures, hand-crafted and heuristic based feature-engineered predictors, or offline imitation models that cannot generate new decision…

  35. arXiv cs.LG TIER_1 English(EN) · Hyesung Jeon, Hyeongju Ha, Jae-Joon Kim ·

    PReCache:通过低秩预计算和中性重构实现多LoRA代理的高效KV缓存共享

    arXiv:2609.34054v2 Announce Type: replace Abstract: Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the growing shared trajectory and constructs its own KV cache, introducing substantial me…

  36. arXiv cs.AI TIER_1 English(EN) · Sara Abdali, Jongwoo Ko, Pashmina Cameron ·

    AttSVD:通过注意力引导的SVD实现Prompt自适应低秩KV缓存压缩

    arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free remedies evict low-importance tokens, an irreversible choice along the sequence ax…

  37. arXiv cs.AI TIER_1 English(EN) · Gleb Molodtsov, Ekaterina Alimaskina, Evgeny Uskov, Artur Zagitov, Aleksandr Beznosikov ·

    Mask-Guided KV Cache Eviction in Block Diffusion Language Models

    arXiv:2610.06996v1 Announce Type: cross Abstract: Block diffusion language models keep a large key-value (KV) cache throughout generation and attend to it at every denoising step, limiting both memory capacity and generation speed. Reducing these costs requires deciding which pas…

  38. Hugging Face Daily Papers TIER_1 English(EN) ·

    DeferKV:重新思考一次性 KV 缓存压缩的驱逐时机

    Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction im…

  39. arXiv cs.AI TIER_1 English(EN) · Minsoo Cheong, Donghyun Son, Sungjoo Yoo ·

    为1位KV缓存压缩定制量化空间

    arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promis…

  40. arXiv cs.LG TIER_1 English(EN) · Inbasekaran S ·

    Page-EntroKV:面向硬件、带熵加权的 KV 缓存逐出策略,用于分组查询注意力

    arXiv:2610.03135v1 Announce Type: new Abstract: Serving long-context autoregressive language models is constrained by the key-value (KV) cache. Most dynamic eviction methods score token importance per query head and choose tokens independently. This fits poorly with grouped-query…

  41. arXiv cs.LG TIER_1 English(EN) · Zihan Teng, Jiayu Zhao, Wentao Ren, Minhao Fan, Tianrui Ma, Song Chen, Weichen Liu ·

    SlimKV:联合 token-特征 KV 缓存压缩与无重构信标注意力

    arXiv:2610.02953v1 Announce Type: new Abstract: Long-context LLM serving is increasingly bottlenecked by KV-cache memory, especially in resource-constrained scenarios. Among existing KV-cache compression strategies, token-wise methods reduce cached states but risk information los…

  42. arXiv cs.CL TIER_1 English(EN) · Haeyong Kang, Chang D. Yoo ·

    跨上下文 KV 缓存重用的预算缓存修复

    arXiv:2610.02233v1 Announce Type: cross Abstract: Cross-context KV-cache reuse predicts a shared segment's keys and values under a new prefix instead of recomputing them, and has been reported to do so without quality loss. We find otherwise, and identify two problems. (1) A hidd…

  43. arXiv cs.AI TIER_1 Deutsch(DE) · Johannes Wesch, Danni Liu, Jan Niehues ·

    KV$^2$: 自我完善的KV缓存

    arXiv:2610.03198v1 Announce Type: new Abstract: The memory footprint of the key-value (KV) cache constrains the practical use of long-context models, and it dominates cost when one prefilled context must later serve many different queries. In this reusable setting, query-agnostic…

  44. arXiv cs.AI TIER_1 English(EN) · Shuxin Liu, Qing Liu, Yi Du, Ou Wu ·

    CORE:KV缓存的覆盖校准和驱逐质量重分布

    arXiv:2610.02235v1 Announce Type: cross Abstract: Long-context decoding is increasingly constrained by key--value (KV) cache memory and bandwidth. Existing fixed-budget compression methods typically separate retention from compensation, while a retention ranking specifies neither…

  45. arXiv cs.AI TIER_1 English(EN) · Han Yu, Wenhui Zhu, Xiwen Chen, Zhipeng Wang, Hejian Sang, Han Shi, Menglin Zhou, Xuanzhao Dong, Minzhou Huang, Rui Cai, Hao Wang, Alborz Geramifard ·

    AvoKV-E:面向长推理的感知负载的KV缓存驱逐

    arXiv:2610.03007v1 Announce Type: cross Abstract: Long-output reasoning shifts the KV-cache bottleneck from the fixed prompt to the generated trace. Existing reasoning-cache eviction methods largely treat cached entries as routing objects, estimating whether an old key will still…

  46. arXiv cs.AI TIER_1 English(EN) · Yiren Zhao, Guanghui Song, Tianrui Qin, Kejiang Ye, Cheng-zhong Xu, Xitong Gao ·

    iS-KV:通过块增量SVD实现在线低秩KV缓存压缩

    arXiv:2610.02815v1 Announce Type: new Abstract: Long chain-of-thought reasoning substantially increases KV-cache memory during autoregressive decoding, as every generated token introduces new key and value states and causes the cache to grow linearly with decoding length. Existin…

  47. arXiv cs.LG TIER_1 English(EN) · Jiawei Lin, Saibo Geng, Thomas Bourgeat ·

    EchoPress:通过虚拟上下文重建实现查询无关的KV缓存修剪

    arXiv:2610.00412v1 Announce Type: new Abstract: KV cache pruning reduces long-context inference memory usage by evicting less important key-value pairs. KVzip estimates importance through context reconstruction: prompting a model to repeat the context chunk by chunk. This achieve…

  48. arXiv cs.LG TIER_1 English(EN) · Snigdha Chandan Khilar ·

    量化递归状态缓存的低差异抖动

    arXiv:2609.39185v1 Announce Type: new Abstract: Mamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this state in low precision saves memory bandwidth, but every rounding error is fed bac…

  49. arXiv cs.LG TIER_1 English(EN) · Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi, Donggyun Kim, Seunghoon Hong ·

    PatchKV:KV缓存的权重空间补偿

    arXiv:2609.39329v1 Announce Type: new Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compression methods reduce the cache through token eviction or approximation, but degrade …

  50. arXiv cs.AI TIER_1 English(EN) · Yikai Wang, Xiao Han, Mengmeng Xu, Juan Camilo Perez, Yiannis Douratsos, Sen He, Zijian Zhou, Fei Zhang, Zhaochong An, Juan-Manuel Perez-Rua, Chen Change Loy, Tao Xiang ·

    具有干净锚点的飞行中 KV 缓存,用于加速自回归视频扩散

    arXiv:2609.32540v2 Announce Type: replace-cross Abstract: Few-step autoregressive video diffusion generates a long video by splitting the video into temporal chunks and generating chunk-by-chunk, each through a short sequence of denoising stages. To memorize chunks that are alrea…

  51. arXiv cs.AI TIER_1 English(EN) · Vincent-Daniel Yun, Woosang Lim, Haneul Yoo, Sungjoo Yoo, Murali Annavaram, Sai Praneeth Karimireddy ·

    异构多智能体大语言模型预填充无关的跨家族KV缓存迁移

    arXiv:2609.32259v2 Announce Type: replace Abstract: Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing th…

  52. arXiv cs.AI TIER_1 English(EN) · Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu ·

    ResidualKV:基于残差的KV缓存压缩,实现高效长上下文推理

    arXiv:2602.08005v2 Announce Type: replace-cross Abstract: Efficient long-context inference faces two coupled bottlenecks: KV-cache memory grows linearly with context length, while attention computation grows quadratically. Existing approaches typically address one at the expense …

  53. Hugging Face Daily Papers TIER_1 English(EN) ·

    量化递归状态缓存的低差异抖动

    Mamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this state in low precision saves memory bandwidth, but every rounding error is fed back into the next update and can accumulate over l…

  54. arXiv cs.AI TIER_1 English(EN) · Xingyu Zhu (Luke), Pu (Luke), Yi, Ziheng Cheng, Ang Lv, Jing Liu, Lexing Ying, Yiyuan Ma, Xin Dong ·

    周期性弱点:分块 KV 缓存压缩的相位敏感性

    arXiv:2609.36322v1 Announce Type: cross Abstract: Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new position…

  55. arXiv cs.LG TIER_1 English(EN) · Jiale Chen, Vage Egiazarian, Eldar Kurti\'c, Torsten Hoefler, Dan Alistarh ·

    WUSH-KV: 具有数据自适应变换的 KV 缓存量化

    arXiv:2609.38121v1 Announce Type: new Abstract: KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low-bit KV-cache quantization. It adapts WUSH, which co…

  56. arXiv cs.AI TIER_1 English(EN) · Zheyu Shen, Guanhua Wang, Dezhan Tu, Mengchi Zhang, Yanjia Li, Adnan Aziz, Chunqiang Tang, Ang Li ·

    ARC-KV:用于基于重建的 KV 缓存压缩的摊销锚点搜索

    arXiv:2609.36835v1 Announce Type: new Abstract: Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. …

  57. arXiv cs.AI TIER_1 English(EN) · Joao Monteiro, Louis B\'ethune, Anastasiia Filippova, Sonia Laguna, David Grangier, Marco Cuturi ·

    KV-Kaizen:学习上下文自适应缓存压缩选择

    arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negatively, since decoding is memory-bound and decode cost …

  58. arXiv cs.LG TIER_1 English(EN) · Nenad Banfic ·

    HeadGuard:低比特VLM KV缓存量化的选择性头部保护

    arXiv:2609.35800v1 Announce Type: new Abstract: Low-bit key-value (KV) cache quantization saves storage but can sharply degrade vision-language model (VLM) accuracy. We introduce HeadGuard, a composable head-protection method that augments a base KV-cache quantizer with a fixed h…

  59. arXiv cs.CL TIER_1 English(EN) · Xinghao Chen, Junnan Dong, Cai Ke, Chak Tou Leong, Haocheng Sun, Keyu Chen, Siyu An, Ruizhi Qiao, Xing Sun, Wenjie Li, Xiaoyu Shen ·

    ATTUNER:通过查询端自适应实现无重计算的KV缓存重用

    arXiv:2609.36722v1 Announce Type: new Abstract: Large language model (LLM) agents repeatedly load reusable content, such as skills, documents, and memory entries, into the current context. Re-encoding this content for every request wastes computation. Position-independent caching…

  60. arXiv cs.CL TIER_1 English(EN) · Zunhai Su, Yuxuan Sun, Jianchao Tan, Tao Zhang, Ruihan Hu, Yuchen Xie, Xunliang Cai, Ngai Wong ·

    QuantMLA:面向低比特MLA KV缓存的函数对齐双路径量化

    arXiv:2609.36760v1 Announce Type: cross Abstract: Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memory still scales linearly with context length and batch size. In this work, we es…

  61. Hugging Face Daily Papers TIER_1 English(EN) ·

    QuantMLA:用于低比特 MLA KV缓存的函数对齐双路径量化

    Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memory still scales linearly with context length and batch size. In this work, we establish a systematic model of MLA's dual-path quan…

  62. arXiv cs.LG TIER_1 English(EN) · Samuel Cestola, Tianxiang Xia, Pengfei Zheng, Weiyan Zheng, Bo Wang, Yi Zhao, Diego Didona ·

    评估KV缓存重用技术的准确性

    arXiv:2609.31415v1 Announce Type: new Abstract: Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV cache reuse techniques rely on measurements that fa…

  63. arXiv cs.LG TIER_1 English(EN) · Tianfang Xie, Wei Zhu ·

    超越平均注意力:面向 KV 缓存驱逐的多样性感知、层级式评分

    arXiv:2609.30738v1 Announce Type: cross Abstract: KV cache eviction methods such as SnapKV and PyramidKV rank tokens solely by mean attention over a small observation window. We study a unified score, $\mu_i+\lambda_1\sigma_i+\lambda_2\mathrm{corr}(i,S)$, adding attention dispers…

  64. arXiv cs.AI TIER_1 English(EN) · Zihan Wang, Cheng Tang, Lei Gong, Chao Wang, Wenqi Lou, Teng Wang, Xuehai Zhou ·

    ActKV:通过动作引导的KV缓存管理实现高效LLM代理

    arXiv:2609.31395v1 Announce Type: cross Abstract: Agentic LLM inference accumulates long KV caches across iterative observation-reasoning-action loops, imposing substantial memory overhead and limiting serving throughput. Existing compression methods emphasize overall output qual…

  65. arXiv cs.LG TIER_1 English(EN) · Tejinder Singh ·

    KV Cache 成为新的内存墙

    arXiv:2609.30854v1 Announce Type: cross Abstract: Autoregressive LLM inference at long context is bounded by memory bandwidth, not arithmetic throughput, and the binding resource shifts from model weights to the Key-Value (KV) cache as sequence length grows. For Llama-3-70B in BF…

  66. arXiv cs.LG TIER_1 English(EN) · Yuhang Cao, Yanzhou Mu, Chunrong Fang, Zhenyu Chen ·

    CacheReforge: 演进式适配器下陈旧 KV 缓存的有界恢复

    arXiv:2609.30884v1 Announce Type: new Abstract: Large language models rely on KV caching to reduce repeated prefill computation in long context and interactive applications. As lightweight adapters evolve, cached states reflect earlier versions, so stale reuse distorts current mo…

  67. Hugging Face Daily Papers TIER_1 English(EN) ·

    WUSH-KV:具有数据自适应变换的KV缓存量化

    KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low-bit KV-cache quantization. It adapts WUSH, which constructs a data-aware transform from the second-…

  68. Hugging Face Daily Papers TIER_1 English(EN) ·

    异构多智能体大语言模型预填充无关的跨家族KV缓存迁移

    Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing the sender's key-value (KV) cache avoids this redundan…

  69. Hugging Face Daily Papers TIER_1 English(EN) ·

    QuantWM:视频世界模型的时序一致性2位KV缓存量化

    Video world models achieve long-range temporal consistency by storing KV cache during generation, but the growing cache makes KV cache memory a major deployment bottleneck, which motivates low-bit quantization study for efficiency. Existing 2-bit KV cache quantization methods can…

  70. Hugging Face Daily Papers TIER_1 English(EN) ·

    周期性弱点:分块 KV 缓存压缩的相位敏感性

    Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token's phase, or its position re…

  71. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Murali Annavaram ·

    异构多智能体大语言模型预填充无关的跨家族KV缓存迁移

    Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing the sender's key-value (KV) cache avoids this redundan…

  72. Hugging Face Daily Papers TIER_1 English(EN) ·

    具有清晰锚点的飞行中 KV 缓存,用于更快的自回归视频扩散

    Few-step autoregressive video diffusion generates a long video by splitting the video into temporal chunks and generating chunk-by-chunk, each through a short sequence of denoising stages. To memorize chunks that are already generated, previous methods reconstruct a clean or less…

  73. arXiv cs.CV TIER_1 English(EN) · David Dobre, Leo Schwinn, Gauthier Gidel, Spandana Gella, Perouz Taslakian, Pierre-Andr\'e No\"el ·

    视觉记忆攻击可穿透 KV 缓存

    arXiv:2610.09027v1 Announce Type: cross Abstract: Modern language model systems operate autonomously over increasingly long contexts containing untrusted text and images. Can an adversarial input continue to steer a model even after that input is removed from its context? We show…

  74. HN — claude cli stories TIER_1 English(EN) · murtazayusuf ·

    Show HN:KiviDB,一个更快的 Redis 兼容内存数据库和向量缓存

  75. Pandaily TIER_1 English(EN) · [email protected] (Pandaily) ·

    ByteDance 种子论文发现分块 KV 缓存压缩中的“相位敏感性”盲点

    A ByteDance Seed paper finds retrieval accuracy in chunk-compressed models such as DeepSeek-V4 can swing by up to 40 points depending on where a token sits.

  76. Towards AI TIER_1 English(EN) · Soumil Gada ·

    KV Cache 只是会话状态:面向后端工程师的解耦推理

    <figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3XqwGL6tcHpQQ6OEYhYe2g.png" /></figure><h4>Prefill, decode, and why routing an LLM request is closer to routing a shopping cart than a stateless API call</h4><p><em>Disclosure: I drafted parts of this article wit…

  77. Medium — Claude tag TIER_1 English(EN) · Automation Labs ·

    Claude 代码空闲压缩(10 月 14 日):54 分钟缓存数学

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@automation.labs/claude-code-idle-compaction-oct-14-the-54-minute-cache-math-04546cbdb7fc?source=rss------claude-5"><img src="https://cdn-images-1.medium.com/max/2600/0*4jCxgbJhtX1na53P" width=…

  78. Medium — MLOps tag TIER_1 English(EN) · Krishna Fattepurkar ·

    KV 缓存内存、放置、重用以及为什么 1.7% 的命中率会导致您损失 46 倍的 TTFT 系列文章 3

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@krishnafattepurkar/the-kv-cache-memory-placement-reuse-and-why-a-1-7-hit-rate-costs-you-46-ttft-series-article-3-536d6d52cb9b?source=rss------mlops-5"><img src="https://cdn-images-1.medium.com…

  79. dev.to — LLM tag TIER_1 English(EN) · ai maya ·

    Ling-3.1-flash与7:1混合模型:560B模型在1M token下需要多少KV缓存?

    <h2> TL;DR </h2> <p>Ling-3.1-flash (MIT license, 560B total, 25B active, weights on Hugging Face since 2026-10-13) replaces most of its attention layers with linear attention: 63 Kimi Delta Attention (KDA) layers and only 9 gated Multi-head Latent Attention (MLA) layers, a 7:1 ra…

  80. r/MachineLearning TIER_1 English(EN) · /u/atronos_kronios ·

    GPT-2 KV缓存从零开始:正确性、延迟和内存测量 [P]

    <!-- SC_OFF --><div class="md"><p>I built GPT-2 small and a KV cache in PyTorch as a learning project, then compared the mathematical savings with measured latency and memory. I have published the implementation and a detailed write-up, and would appreciate feedback on the method…

  81. dev.to — LLM tag TIER_1 English(EN) · Kai ·

    长上下文为何昂贵:KV缓存的账单

    <p>Your chat history can weigh more than the model that reads it. Llama 3.1 8B is a 16 GB model. Give it a 1M-token prompt and its KV cache alone needs 128 GB.</p> <h2> Why the cache exists </h2> <p>Transformers have no memory of their own. RNNs kept one small hidden state, const…

  82. dev.to — LLM tag TIER_1 English(EN) · Felipe 0liveira ·

    KV缓存现在是瓶颈——1比特量化、MoE掉队者和按秒计费的GPU

    <p>Welcome to this week's LLM Inference Digest, covering roughly September 30 – October 7, 2026 across arXiv (cs.LG, cs.DC), the vLLM blog, Hugging Face blog, Together AI blog, Modal blog, and Latent Space.</p> <h2> 🔥 Highlights </h2> <ol> <li><p><strong><a href="https://arxiv.or…

  83. dev.to — LLM tag TIER_1 English(EN) · AI Tech News ·

    LLM服务中的KV Cache量化:FP8与INT8的权衡,隐藏的config.json陷阱,以及如何公平衡量

    <h2> TL;DR </h2> <p>The KV cache is often the real memory ceiling in LLM serving, not the weights. Quantizing it to FP8 or INT8 roughly halves KV memory, which lets you serve longer contexts or more concurrent requests on the same GPU. The catch: the quality impact is real but sm…

  84. dev.to — LLM tag TIER_1 English(EN) · threataft ·

    月饼大规模披露 — KV缓存传输引擎中存在 CVSS 9.8 任意内存读/写漏洞

    <p>A crafted TCP packet to Mooncake's transfer data port is enough to read and write arbitrary process memory — no authentication required.</p> <p>CVE-2026-103764 (CVSS 9.8) is an untrusted pointer dereference in ServerSession::readHeader. The readHeader function trusts attacker-…