KV cache
PulseAugur coverage of KV cache — every cluster mentioning KV cache across labs, papers, and developer communities, ranked by signal.
- used by large-language models 90%
- used by GQA 90%
- uses TurboQuant 90%
- used by Grouped Query Attention 90%
- used by TurboQuant 90%
- used by Prompt Caching for Token Efficiency 90%
- used by Mingxin Technology 90%
- instance of Kivi 90%
- instance of SnapKV 90%
- used by PagedAttention 90%
- developed by Oscar 90%
- used by Oscar 90%
27 day(s) with sentiment data
-
AI agents vulnerable to KV cache poisoning attacks
A new technical analysis reveals significant security vulnerabilities in stateful AI agents, particularly concerning the Key-Value (KV) cache. Attackers can exploit this memory to perform indirect prompt injections and …
-
AI systems' "zero data retention" claims often omit critical data persistence points
The concept of "zero data retention" in AI systems is often narrowly applied, typically covering only the non-persistence of request and response content to durable storage or its exclusion from training pipelines. Howe…
-
ImpactHO improves edge LLM handover with importance-aware KV cache transfer
Researchers have developed a novel method called ImpactHO to improve the efficiency of transferring Key-Value (KV) caches between edge nodes for Large Language Models (LLMs). This approach prioritizes the most important…
-
LLM Server Memory Accounting: Reserve for Max Output, Not Just Prompt
A technical post explains how to prevent LLM servers from crashing due to memory issues by correctly accounting for KV cache usage. The key insight is that memory reservations should be based on the maximum potential ou…
-
New MISA-T policy boosts RL rollout efficiency for LLMs
Researchers have developed MISA-T, a new routing-layer admission policy designed to optimize the scheduling of mixed reinforcement learning (RL) rollouts for large language models (LLMs). This policy addresses the chall…
-
Compute rental contracts need specific clauses for AI workloads
This article highlights three critical but often overlooked clauses in compute rental contracts for AI workloads: bandwidth, storage, and failure duration. It emphasizes that network bandwidth is crucial for large model…
-
New 'Full-Bandwidth Transformer' Enhances AI Model Feedback Channels
Researchers have introduced a novel architecture called the "full-bandwidth transformer" that enhances the vertical feedback channel in autoregressive transformers. This new design incorporates "latent feedback," where …
-
VoxZip framework slashes audio LLM KV cache needs by 20x
Researchers have developed VoxZip, a novel two-stage framework designed to compress the KV cache for long-context audio inference in Speech Large Language Models. This method uses Automatic Speech Recognition (ASR) tran…
-
LLM Admission Control Crucial for Self-Hosted Stability
Self-hosting large language models (LLMs) can lead to crashes under heavy load due to the KV cache, which consumes significant GPU memory per request and grows with context length and concurrency. This memory usage, rat…
-
KV Cache Emerges as LLM Bottleneck, Driving Attention Variant Innovations
The KV cache, a critical component in autoregressive decoding for LLMs, is identified as the primary bottleneck for frontier models in 2026. Its size grows linearly with context length and batch size, making it the domi…
-
New tools enable LLM fine-tuning on low-spec hardware
New tools and techniques are emerging to enable fine-tuning and running large language models (LLMs) on consumer-grade hardware. Soup CLI, an open-source Python tool, utilizes layer streaming to fine-tune an 8B LLM on a…
-
Compressed Sensing Unsuitable for LLM Inference Storage Compression
Compressed sensing is not a suitable method for compressing KV cache data during LLM inference due to the data's lack of sparsity and the need for deterministic, lossless operations. Instead, practical improvements in i…
-
KV Cache Prefetching Slashes LLM Inference Latency
A new prefetching strategy for KV Cache data has been developed, significantly reducing storage latency during large model inference. This method, tested on the Mingxin FX100 with a 480B model, improves inference throug…
-
New QEvict system improves LLM long-context decoding with recoverable KV cache
Researchers have introduced QEvict, a novel three-tier KV-cache management system designed to improve long-context decoding in autoregressive large language models. Unlike existing methods that permanently discard token…
-
AI infrastructure evolves to integrate storage for LLM inference
The AI infrastructure landscape is shifting from solely focusing on GPU compute to a more integrated approach involving compute, networking, memory, and storage. This evolution is driven by the demands of large language…
-
New WorldTrace framework improves visual memory in video models
Researchers have developed WorldTrace, a novel framework designed to enhance visual persistence in video world models. This new approach addresses limitations in existing models that struggle to recall information beyon…
-
AMD's top AI engineers concentrated in Shanghai for ROCm development
A significant portion of AMD's top AI engineering talent, including key teams focused on MoRI, KV-cache offloading, and pooling, is located in Shanghai. This concentration of expertise is crucial for developing core com…
-
SK Hynix, SanDisk unveil High Bandwidth Flash for AI inference memory wall
SK Hynix and SanDisk have collaborated to develop High Bandwidth Flash (HBF), a new memory tier designed to address the memory wall challenges in AI inference. HBF places large-capacity NAND flash memory close to variou…
-
KV Cache Transfer Speeds Up LLM Inference by Up to 25x
Researchers have developed a method to transfer KV caches between different-sized language models within the same family, significantly speeding up inference when switching models. This technique involves fitting a line…
-
LLM routing strategy boosts KV cache hits, slashes latency
A new routing strategy for large language models (LLMs) aims to improve efficiency by intelligently managing the KV cache. Unlike traditional round-robin load balancers that distribute requests blindly, this KV-cache-aw…