KV caches
PulseAugur coverage of KV caches — every cluster mentioning KV caches across labs, papers, and developer communities, ranked by signal.
-
6 Architectural Shifts to Optimize LLM Pipelines for Cost and Latency
The article proposes six architectural shifts to optimize large language model (LLM) pipelines by reducing token costs and latency. It advocates for implementing strict retrieval-augmented generation (RAG) with vector d…
-
BinaryPC offers training-free sparse attention for efficient LLM decoding
Researchers have developed BinaryPC, a novel sparse attention mechanism designed to improve the efficiency of long-context large language models. This training-free method uses binary principal components to create comp…
-
Llama-server bug discarded KV caches, fix restores fast state restores
A bug in llama-server caused it to discard restored KV caches, forcing a full re-prefill and significantly increasing processing time. The issue stemmed from the server's state saving mechanism, which serialized token d…
-
InfoFlow KV improves retrieval-augmented generation for long contexts
Researchers have developed InfoFlow KV, a novel method for improving retrieval-augmented generation (RAG) in large language models. This technique addresses the bottleneck of prefilling large retrieved contexts during i…
-
TurboQuant compresses AI vectors to 2-4 bits without accuracy loss
A new method called TurboQuant has been developed to compress AI vectors, such as those in KV caches and attention keys, to as few as 2-4 bits per number without sacrificing accuracy. This technique relies on the princi…