Transformer attention
PulseAugur coverage of Transformer attention — every cluster mentioning Transformer attention across labs, papers, and developer communities, ranked by signal.
-
New research optimizes transformer attention with Mathematics of Arrays
A new research paper details a method for optimizing transformer attention inference using the Mathematics of Arrays (MoA). The paper presents four memory-efficient artifacts, including a single-query decode DNF that al…
-
RayPE encoding boosts 3D awareness in video generation models
Researchers have developed RayPE, a novel positional encoding method for video diffusion transformers that enhances 3D awareness. Unlike existing methods that use camera grid coordinates, RayPE incorporates 6D Plucker c…
-
Transformer attention shows deficient executive control, study finds
A new research paper explores the limitations of transformer attention mechanisms, specifically focusing on their "executive control" capabilities. The study, published in PNAS Nexus, suggests that while transformers ex…
-
FlashSinkhorn solver accelerates optimal transport on GPUs
Researchers have developed FlashSinkhorn, a new GPU-accelerated solver for entropic optimal transport (EOT) that significantly reduces memory input/output operations. By rewriting stabilized log-domain Sinkhorn updates …
-
Energy-Gated Attention enhances Transformer models by prioritizing salient tokens
Researchers have introduced Energy-Gated Attention (EGA), a novel mechanism designed to improve transformer models by focusing on spectrally salient tokens. This approach mimics principles from fluid dynamics, prioritiz…