Kronecker Product Attention offers 10x KV cache compression for LLMs

Researchers have introduced Kronecker Product Attention (KPA), a novel method for compressing the key-value (KV) cache in autoregressive models. KPA represents KV activations using sums of Kronecker products of compact matrix factors, which strictly generalizes existing techniques like Tensor Product Attention. This approach aims to reduce the memory footprint that limits efficient generation without sacrificing model expressivity or requiring complex positional encoding machinery. Empirically, KPA has demonstrated competitive or superior performance across various model scales and benchmarks while achieving up to a 10x KV cache compression. AI

IMPACT This method could significantly reduce the memory requirements for running large language models, potentially enabling more efficient deployment and accessibility.

RANK_REASON The item is a research paper detailing a new method for KV cache compression in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Grouped-Query Attention
• Hugging Face
• Kronecker Product Attention
• KV cache
• Multi-Head Latent Attention
• Multi-Query Attention
• RoPE
• Tensor Product Attention

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →