KV cache
PulseAugur coverage of KV cache — every cluster mentioning KV cache across labs, papers, and developer communities, ranked by signal.
- uses TurboQuant 90%
- used by GQA 90%
- used by TurboQuant 90%
- used by Prompt Caching for Token Efficiency 90%
- used by Mingxin FX100 90%
- instance of SnapKV 90%
- used by Oscar 90%
- instance of Kivi 90%
- developed by Oscar 90%
- used by Grouped Query Attention 90%
- instance of ruler 90%
- used by graphics processing unit 70%
17 day(s) with sentiment data
-
Huawei launches OceanStor M900 for AI inference with petabyte KV cache · 2 sources tracked
Huawei has introduced the OceanStor M900, a new AI memory storage solution designed for hyperscale inference. This system features a petabyte-scale key-value (KV) cache, aiming to enhance AI infrastructure for applicati…
-
Withdrawn paper proposed KV cache compression for LLM alignment
A research paper, since withdrawn by its author Rui Zhu, explored methods to compress the KV cache in Large Language Models (LLMs) during post-training alignment. The study aimed to address the significant memory overhe…
-
HyQuant framework optimizes LLM attention with hybrid-precision quantization
Researchers have developed HyQuant, a novel hybrid-precision quantization framework designed to improve the efficiency of Large Language Model (LLM) attention mechanisms. This method quantizes most attention states to l…
-
AI agents to report tool progress for better performance
Researchers have developed a new method for AI agents to better manage their progress when using external tools. Current systems often guess how long a tool will take, leading to inefficient use of GPU memory. The propo…
-
Prompt Caching Slashes LLM Costs Up to 75% by Reusing KV Cache
Prompt caching is a technique that can significantly reduce the cost of using large language models by reusing computed states, known as the KV cache. This method is most effective when static content, such as system pr…
-
New research compares LLM knowledge injection methods
A new research paper explores methods for large language models (LLMs) to access new information beyond their pre-training data. The study compares representation-based (KV-cache) and parametric (fine-tuning) adaptation…
-
KV Cache Placement Strategies Explored for LLM Memory Efficiency
A new research paper explores optimal placement strategies for KV caches across different memory tiers (GPU HBM, CPU DRAM, SSD) to manage scarce GPU memory. The study, conducted using a discrete event simulator, found t…
-
AI models offload memory to CPUs to boost performance
Large language models are facing memory challenges as AI agents require extensive context, leading to large KV caches that strain GPU memory. To address this, a new approach shifts memory management from GPUs to CPUs, u…
-
KV Cache Management: Optimizing GPU Memory for LLMs
The KV cache, a critical component for large language models, expands with each token processed, potentially consuming significant GPU memory. This article explores four adjustable parameters that can help manage KV cac…
-
AgentKV improves LLM efficiency with phase-aware KV eviction
Researchers have developed AgentKV, a new method for managing KV cache in agentic Large Language Models (LLMs). AgentKV addresses the issue that traditional KV eviction methods, which rely on recent tokens, fail to acco…
-
New research optimizes KV cache usage for LLMs, improving efficiency and accuracy
Recent research explores methods to optimize KV cache usage in large language models, particularly for long contexts and agentic systems. One paper proposes a budgeted repair strategy for stale KV caches after document …
-
Visual KV-cache retention is task-inert, new paper finds
A new research paper challenges the assumption that visual key-value (KV) caches in vision-language models retain task-relevant information. The study found that the amount of visual content retained in the KV cache is …
-
LRU Caching Proves Surprisingly Resilient Against Advanced AI Cache Methods
A recent article suggests that Least Recently Used (LRU) caching strategies are more effective than previously understood, even when compared to advanced KV-cache techniques used in AI models. The author highlights that…
-
CLAUDE.md file goes viral; GPT-6 Astra introduces looped transformers
A single file named CLAUDE.md has gone viral, offering lessons in AI discipline for branding agencies. Separately, a new model called GPT-6 Astra has been introduced, featuring looped transformers that effectively doubl…
-
REVA framework optimizes RAG systems for efficiency and quality
Researchers have developed REVA, a novel framework designed to enhance the efficiency of retrieval-augmented generation (RAG) systems. REVA addresses the challenges of increased latency and memory usage associated with …
-
KV-Cache Side Channel Reliability Collapses Under LLM Serving Load
Researchers have investigated the reliability of KV-cache timing side channels in multi-tenant LLM serving environments. Their experiments revealed that contention from multiple users significantly degrades the reliabil…
-
New research tackles LLM efficiency with NVM quantization and on-device power scaling
Two new research papers explore methods to improve the efficiency of large language models (LLMs). The first paper, "Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding," proposes a …
-
New BIO-MEMART framework uses biometrics for secure LLM agent memory
Researchers have developed BIO-MEMART, a novel framework for managing KV cache memory in multi-user LLM agents. This system enhances security by incorporating biometric authentication to control access to shared memory …
-
LLM routing methods improve efficiency and reduce latency · 2 sources tracked
A new research paper introduces HeRo (History-Aware Routing), a dynamic routing framework for large language models that uses a memory mechanism to maintain routing state across model depth. This approach, which aggrega…
-
LLM inference engines optimize prompt processing with prefix caching
Large Language Models (LLMs) often recompute the same initial prompt tokens repeatedly, leading to inefficiency. This article explains that the KV cache, which stores intermediate states during token generation, is the …