PyTorch FlashAttention

New 'Relation' primitive challenges Attention in language models

Researchers have introduced "Relation," a novel token-mixing primitive for language models that organizes pairwise evidence into explicit Self and Exchange relations before deriving information flow. This approach yield…