KV cache
PulseAugur coverage of KV cache — every cluster mentioning KV cache across labs, papers, and developer communities, ranked by signal.
- instance of alphaXiv 90%
- uses TurboQuant 90%
- used by TurboQuant 90%
- used by Mingxin FX100 90%
- used by Prompt Caching for Token Efficiency 90%
- instance of SnapKV 90%
- instance of ruler 90%
- instance of Kivi 90%
- developed by Oscar 90%
- instance of Multi-layer Attention Neural Network for Sentence Semantic Matching 90%
- used by Oscar 90%
- used by vLLM 70%
27 day(s) with sentiment data
-
Edge device RAM for LLMs: Beyond simple subtraction
Running large language models on edge devices with limited RAM requires careful memory management beyond simply subtracting model file size from total device memory. Factors like the KV cache, operating system overhead,…
-
Scratchpad Patching improves byte-level language models
Researchers have introduced Scratchpad Patching (SP), a novel technique for byte-level language models that decouples compute requirements from patch size. This method addresses the trade-off between larger patches (red…
-
New BitDecoding-FP4 method boosts LLM serving throughput with FP4 KV cache
Researchers have developed BitDecoding-FP4, a novel algorithm-system co-design for efficient and accurate native FP4 KV cache decoding in large language models. This approach addresses accuracy drops and throughput degr…
-
AI Model Speed Differences Explained by Chip Architecture and Caching
AI models perform faster on certain chips due to architectural differences and optimization techniques. Key factors include the use of KV cache, which acts as a model's memory to avoid redundant calculations. The way da…
-
On-policy distillation tolerates significant training-inference mismatch
Researchers have investigated the tolerance of on-policy distillation (OPD) for training-inference mismatch, a common issue when using quantized inference and asynchronous rollouts to reduce costs. Their findings indica…
-
Fixing slow LLM responses after restarts with llama.cpp KV cache
A technical explanation addresses the performance degradation experienced with long conversations in local LLM applications like llama.cpp after a server restart. The issue stems from the KV cache, which stores processe…
-
Cloudflare Workers AI leaks customer prompts due to KV cache bug
A bug in Cloudflare's Workers AI service, specifically within the KV cache mechanism for the GLM 5.3 model, inadvertently exposed parts of previous customer prompts to subsequent users. This memory bookkeeping error occ…
-
New methods tackle KV cache compression for visual generation models · 2 sources tracked
Two new research papers introduce methods to compress KV caches for autoregressive visual generation models, addressing memory and computational bottlenecks. VAR-Q focuses on tuning-free quantization by analyzing channe…
-
Long context LLM windows are a billing trick, not a capability boost
The author argues that the perceived intelligence boost from longer context windows in LLMs is often an illusion, driven by billing decisions rather than true capability improvements. While larger windows make short tas…
-
LoopVQ method slashes KV cache memory for looped transformers
Researchers have developed LoopVQ, a novel vector quantization method designed to significantly reduce the memory footprint of the key-value (KV) cache in looped transformers. This technique exploits redundancy across l…
-
KV cache quantization creates LLM safety vulnerabilities, study finds
A recent finding from the ICML 2026 Mechanistic Interpretability Workshop reveals that quantizing the KV cache in large language models can inadvertently reintroduce vulnerabilities. While this optimization significantl…
-
New ACCT method enables efficient KV cache transfer between LLMs
Researchers have developed ACCT (Asymmetric Cross-LLM Context Transfer), a novel method to reduce computational costs in heterogeneous large language model (LLM) deployments. ACCT enables the transfer of KV cache betwee…
-
Dynamical Memory Transformer offers constant cost attention for long sequences
Researchers have introduced the Dynamical Memory Transformer (DMT), a novel hybrid architecture designed to improve how models handle long sequences. Unlike standard transformers that store an explicit history, leading …
-
New attention methods enable LLMs to process millions of tokens
Researchers are developing new attention mechanisms to handle extremely long contexts in large language models, aiming to overcome the quadratic cost of traditional methods. Papers introduce techniques like Recursive Sp…
-
64GB RAM viable for local LLMs with agents, user reports
A user on Reddit's r/LocalLLaMA community has found that 64GB of RAM is no longer a "dead zone" for running local large language models, particularly when utilizing agents for coding tasks. Their experience with a 27B p…
-
Cloudflare GLM-5.3 model leaked tenant KV cache data due to SGLang bug
Cloudflare experienced a data leak between August and September 2026, where fragments of one tenant's KV cache state were exposed in 155 responses from its GLM-5.3 model on Workers AI. This occurred due to a combination…
-
LLM inference bottlenecked by memory bandwidth, not compute
Large language models are primarily bottlenecked by memory bandwidth rather than computational power, a challenge that persists across GPU generations. The "Decode" operation, which involves reading model weights and KV…
-
KV Cache Dominates LLM Inference Costs, Architecture Key to Savings
The KV cache, not compute power, is the primary driver of inference costs for large language models, especially with long contexts. This is because the KV cache grows linearly with context length and is accessed repeate…
-
AI LLM Primer: Understanding Weights, Quantization, and Inference
This article explains fundamental concepts in AI Large Language Models (LLMs), including weights, quantization, and inference. Weights are crucial for determining relationships between tokens, with their quantity often …
-
llama.cpp adds DPAS flash attention kernel for quantized KV cache
The llama.cpp project has released version b11980, introducing a new DPAS flash attention kernel for quantized KV caches. This optimization allows for faster processing of small batches and quantized KV blocks, improvin…