New linearized attention model achieves higher accuracy and lower perplexity

Researchers have developed a linearized version of 2-simplicial attention, which rewrites the trilinear score into an inner product. This new form allows for linear cost in sequence length while maintaining global reach, unlike traditional windowed attention mechanisms. By approximating the sum with positive random features and storing past information in a fixed-size state, this method can be implemented with custom Triton kernels and integrated with Kimi Delta Attention. The resulting model, which omits softmax attention entirely, has demonstrated superior downstream accuracy and improved perplexity on benchmarks like LAMBADA. AI

IMPACT This research introduces a more efficient attention mechanism that could lead to faster and more accurate language models.

RANK_REASON The cluster describes a new academic paper detailing a novel attention mechanism for AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• 2-Simplicial Attention
• Kimi Delta Attention
• lambada
• Triton

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →