BellowsKV method drastically cuts LLM KV cache size with minimal accuracy loss

Researchers have developed BellowsKV, a novel method to compress the key-value cache in large language models, which is a significant bottleneck for GPU serving. Unlike previous methods that quantize or evict tokens within a single layer, BellowsKV maps hidden states from adjacent layers to a shared latent representation. This approach allows layers to access their own keys and values through layer-specific projections from the shared latent. BellowsKV-Quant achieves nearly lossless accuracy at 3.6x compression and 97-98% accuracy at 13-14x compression, significantly outperforming existing cross-layer methods. AI

IMPACT This technique could significantly reduce serving costs and increase the efficiency of large language models.

RANK_REASON The cluster contains a research paper detailing a new method for LLM KV cache compression. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• BellowsKV
• BellowsKV-Evict
• BellowsKV-Quant
• graphics processing unit
• Hugging Face
• Llama-3.1:8b
• mistral:7b
• qwen2.5:7b

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →