Blurry Window Attention improves Transformer efficiency for long contexts

Researchers have introduced Blurry Window Attention (BLA), a novel method designed to improve the efficiency of Transformer language models in handling long contexts. BLA addresses the quadratic complexity and growing KV cache size limitations of standard Softmax Attention by reconstructing a blurry KV history from a frequency window using Dirichlet kernels. This approach offers state efficiency improvements over Sliding Window Attention and maintains competitive performance with other linear attention models on tasks requiring information retrieval. AI

IMPACT Introduces a more efficient attention mechanism for handling long sequences in language models.

RANK_REASON The cluster contains an academic paper detailing a new method for improving Transformer efficiency. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Attention with Bounded-memory Control
• Blurry Window Attention
• Dirichlet kernels
• Linear Attention
• Multi-Query Associate Recall
• RegBench
• Sliding Window Attention
• Softmax Attention
• State-Space Models
• Transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →