研究人员正在开发新的方法来压缩大型语言模型（LLM）中的键值（KV）缓存，以减少内存使用并提高推理效率。AnchorKV通过偏向于不保留有害提示的token来关注安全性，而PolyKV通过对不同的Transformer层应用不同的策略和预算来优化压缩。Tangram在服务框架中实现了实用的非均匀KV缓存压缩，而BACON通过结合观察窗口注意力和最后查询证据来增强多模态KV缓存压缩。此外，TurboQuant和OSCAR代表了KV缓存量化的新方法，其中TurboQuant提供了一种数据无关的方法，OSCAR则是一种面向部署的、注意力感知的系统。
AI
arXiv:2606.17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment. Since sca…
Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment. Since scaling pre-trained language models improves downstre…
arXiv:2606.15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference. Existing approaches, however, typically apply a single compression policy and a uniform cache budget across all transfo…
arXiv:2606.14782v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong vision-language reasoning, but long visual contexts enlarge the KV cache and increase decoding latency. Existing compression methods rely on observation window attention for …
Multi-turn large language model serving faces memory constraints due to growing key-value cache, but a structured approach to non-uniform compression enables significant throughput improvements through static budget allocation and optimized memory management.
<p>The KV cache now outweighs model weights at long context. Here's how TurboQuant, OSCAR, and EpiCache each attack that memory bottleneck — and why they're more complementary than competitive.</p> <p>The post <a href="https://www.marktechpost.com/2026/06/18/the-kv-cache-compress…
Three new methods for compressing the LLM KV cache have emerged in 2026, aiming to cut the memory cost of long-context processing. TurboQuant, OSCAR, and EpiCache each take different approaches to reduce KV cache size while preserving model quality. https://www. marktechpost.com/…
Medium — MLOps tag
TIER_1English(EN)·Carnot Research Pvt. Ltd.·
