Keyless Attention mechanism halves KV cache and boosts transformer efficiency

Researchers have introduced Keyless Attention, a novel attention mechanism for transformers that eliminates the key projection entirely, operating solely on queries and values. This approach results in a Value-Only Cache that halves KV cache memory and access overhead compared to standard attention, while maintaining or improving decode throughput. The mechanism also enables Depth-m Attention Factorization, with experiments showing that Keyless Attention matches or surpasses standard QKV attention in perplexity across multiple models and architectures, and outperforms on commonsense reasoning benchmarks. AI

IMPACT This novel attention mechanism could significantly reduce computational costs and memory requirements for large language models, potentially accelerating inference and enabling larger context windows.

RANK_REASON The cluster contains a research paper detailing a novel technical approach for improving transformer efficiency. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Depth-m Attention Factorization
• GPT-2
• Keyless Attention
• KV cache
• Llama 3.2
• Pythia
• QKV attention
• Queries
• Qwen2
• Transformers
• Values

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →