New 'Relation' primitive challenges Attention in language models

Researchers have introduced "Relation," a novel token-mixing primitive for language models that organizes pairwise evidence into explicit Self and Exchange relations before deriving information flow. This approach yields several variants, including Full Relation, FlashRelation, Linear Relation, and Hybrid Relation, along with a Relation Cache. Experiments show that Full Relation achieves lower validation NLL than standard Multi-Head Attention (MHA) across various model scales, while FlashRelation offers significant speedups compared to materialized Full Relation and approaches PyTorch FlashAttention throughput. Hybrid Relation demonstrates strong language-modeling quality by using a majority of Linear Relation layers. AI

IMPACT Introduces a new architectural primitive that could offer performance and efficiency gains over standard attention mechanisms in LLMs.

RANK_REASON Research paper introducing a novel technical primitive for language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• FlashRelation
• Full Relation
• Hugging Face
• Hybrid Relation
• PyTorch FlashAttention
• Relation
• Relation Cache

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →