Brief

Researchers have introduced Energy-Gated Attention (EGA), a novel mechanism designed to improve transformer models by focusing on spectrally salient tokens. This approach mimics principles from fluid dynamics, prioritizing information-dense tokens that hold a disproportionate amount of spectral energy. EGA achieves significant validation loss improvements on datasets like TinyShakespeare and Penn Treebank with minimal parameter overhead and no added computational cost. AI

Researchers have developed FlashSinkhorn, a new GPU-accelerated solver for entropic optimal transport (EOT) that significantly reduces memory input/output operations. By rewriting stabilized log-domain Sinkhorn updates to mimic the normalization process in transformer attention, FlashSinkhorn enables fused kernels that stream data through on-chip SRAM. This approach achieves substantial speedups, up to 32x for forward passes and 161x end-to-end, compared to existing methods on A100 GPUs for tasks like point-cloud OT. AI