OpenAttention method retains 99% accuracy at 1% read rate

Researchers have introduced OpenAttention, a novel method for sparse attention in large language models that significantly reduces computational cost while maintaining high accuracy. The technique utilizes "tail proxies" to represent the unread portions of the key-value cache, improving upon previous methods that either discarded this information or sampled it inefficiently. OpenAttention achieves 99% of dense attention accuracy at a mere 1% read rate on the RULER-32K HARD benchmark using Llama-3.1-8B, and its custom attention kernel offers a tenfold speedup in the decode attention stage. AI

IMPACT Reduces computational costs for LLMs, potentially enabling faster inference and training with less hardware.

RANK_REASON The item is an arXiv preprint detailing a new method for sparse attention in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• ErrorGate
• Hugging Face
• Llama-3.1:8b
• OpenAttention
• QGC
• qgs
• Query-Geometry Clustering
• Query-Geometry Selector
• RULER-32K HARD

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →