Kernelized Linear Attention Boosts Model Capacity and Throughput

Researchers have developed Kernelized Linear Attention Activations (KATA), a new framework designed to overcome the capacity limitations of linear attention models. KATA utilizes symmetric cones and rank-one positive semi-definite features to improve associative recall and state capacity without increasing parameters. Implemented with fused Triton kernels, KATA demonstrates significantly higher throughput than FlashAttention-2, particularly for long sequences, and shows competitive performance against models like Gated DeltaNet on tasks requiring long-range dependencies and precise recall. AI

IMPACT Introduces a novel attention mechanism that significantly enhances model capacity and inference speed, potentially impacting the efficiency of large language models.

RANK_REASON The item is an academic paper detailing a new method for improving attention mechanisms in machine learning models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• FlashAttention-2
• Gated DeltaNet
• KATA
• Kernelized Linear Attention
• Triton

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →