Transformers' expressive power explained by new measure-theoretic framework

Researchers have introduced a new measure-theoretic framework to understand the expressive power of Transformer architectures in modeling contextual relations. This framework connects standard softmax attention to entropy-regularized optimal transport, viewing attention as a normalized affinity function. The study establishes a universal approximation theorem, demonstrating that Transformers can approximate arbitrary contextual relation rules, with the normalization method influencing the representation of these relations. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Provides a theoretical foundation for Transformer capabilities, potentially guiding future architectural improvements.

RANK_REASON Academic paper introducing a new theoretical framework for understanding Transformer architectures.

Read on arXiv cs.LG →

• Transformer
• Softmax Attention
• Demián Fraiman
• arXiv
• Hugging Face

• paper
• other