KV cache
PulseAugur coverage of KV cache — every cluster mentioning KV cache across labs, papers, and developer communities, ranked by signal.
- used by large-language models 90%
- used by TurboQuant 90%
- used by GQA 90%
- uses TurboQuant 90%
- used by Grouped Query Attention 90%
- used by Prompt Caching for Token Efficiency 90%
- instance of SnapKV 90%
- used by Oscar 90%
- used by PagedAttention 90%
- used by Mingxin Technology 90%
- developed by Oscar 90%
- instance of Kivi 90%
28 day(s) with sentiment data
-
ImpactHO improves edge LLM handover with importance-aware KV cache transfer
Researchers have developed a novel method called ImpactHO to improve the efficiency of transferring Key-Value (KV) caches between edge nodes for Large Language Models (LLMs). This approach prioritizes the most important…
-
LLM Server Memory Accounting: Reserve for Max Output, Not Just Prompt
A technical post explains how to prevent LLM servers from crashing due to memory issues by correctly accounting for KV cache usage. The key insight is that memory reservations should be based on the maximum potential ou…
-
Compute rental contracts need specific clauses for AI workloads
This article highlights three critical but often overlooked clauses in compute rental contracts for AI workloads: bandwidth, storage, and failure duration. It emphasizes that network bandwidth is crucial for large model…
-
New 'Full-Bandwidth Transformer' Enhances AI Model Feedback Channels
Researchers have introduced a novel architecture called the "full-bandwidth transformer" that enhances the vertical feedback channel in autoregressive transformers. This new design incorporates "latent feedback," where …
-
VoxZip framework slashes audio LLM KV cache needs by 20x
Researchers have developed VoxZip, a novel two-stage framework designed to compress the KV cache for long-context audio inference in Speech Large Language Models. This method uses Automatic Speech Recognition (ASR) tran…
-
LLM Admission Control Crucial for Self-Hosted Stability
Self-hosting large language models (LLMs) can lead to crashes under heavy load due to the KV cache, which consumes significant GPU memory per request and grows with context length and concurrency. This memory usage, rat…
-
KV Cache Emerges as LLM Bottleneck, Driving Attention Variant Innovations
The KV cache, a critical component in autoregressive decoding for LLMs, is identified as the primary bottleneck for frontier models in 2026. Its size grows linearly with context length and batch size, making it the domi…
-
New tools enable LLM fine-tuning on low-spec hardware
New tools and techniques are emerging to enable fine-tuning and running large language models (LLMs) on consumer-grade hardware. Soup CLI, an open-source Python tool, utilizes layer streaming to fine-tune an 8B LLM on a…
-
Compressed Sensing Unsuitable for LLM Inference Storage Compression
Compressed sensing is not a suitable method for compressing KV cache data during LLM inference due to the data's lack of sparsity and the need for deterministic, lossless operations. Instead, practical improvements in i…
-
KV Cache Prefetching Slashes LLM Inference Latency
A new prefetching strategy for KV Cache data has been developed, significantly reducing storage latency during large model inference. This method, tested on the Mingxin FX100 with a 480B model, improves inference throug…
-
New QEvict system improves LLM long-context decoding with recoverable KV cache
Researchers have introduced QEvict, a novel three-tier KV-cache management system designed to improve long-context decoding in autoregressive large language models. Unlike existing methods that permanently discard token…
-
AI infrastructure evolves to integrate storage for LLM inference
The AI infrastructure landscape is shifting from solely focusing on GPU compute to a more integrated approach involving compute, networking, memory, and storage. This evolution is driven by the demands of large language…
-
New WorldTrace framework improves visual memory in video models
Researchers have developed WorldTrace, a novel framework designed to enhance visual persistence in video world models. This new approach addresses limitations in existing models that struggle to recall information beyon…
-
AMD's top AI engineers concentrated in Shanghai for ROCm development
A significant portion of AMD's top AI engineering talent, including key teams focused on MoRI, KV-cache offloading, and pooling, is located in Shanghai. This concentration of expertise is crucial for developing core com…
-
SK Hynix, SanDisk unveil High Bandwidth Flash for AI inference memory wall
SK Hynix and SanDisk have collaborated to develop High Bandwidth Flash (HBF), a new memory tier designed to address the memory wall challenges in AI inference. HBF places large-capacity NAND flash memory close to variou…
-
KV Cache Transfer Speeds Up LLM Inference by Up to 25x
Researchers have developed a method to transfer KV caches between different-sized language models within the same family, significantly speeding up inference when switching models. This technique involves fitting a line…
-
LLM routing strategy boosts KV cache hits, slashes latency
A new routing strategy for large language models (LLMs) aims to improve efficiency by intelligently managing the KV cache. Unlike traditional round-robin load balancers that distribute requests blindly, this KV-cache-aw…
-
New ReCo framework cuts reasoning model costs by up to 65%
Researchers have developed a new framework called ReCo (Reward-Coordinated Compression) to improve the efficiency of large reasoning models. This method addresses the issue of "overthinking" in models that use long chai…
-
Mingxin FX100 storage solution accelerates video inference, reducing latency
Mingxin's FX100 storage solution addresses latency bottlenecks in real-time video inference, which are often caused by storage and data path limitations rather than GPU compute. The system employs a tiered KV cache appr…
-
LLM Deployment: Prioritize VRAM Over GPU Specs for Efficiency
When deploying large language models, prioritizing VRAM requirements over specific GPU models is crucial for efficient infrastructure planning. Developers should first determine the necessary VRAM by considering factors…