研究人员开发了两种新方法来提高vision transformer中注意力机制的效率。QFlash专注于为FlashAttention实现纯整数运算，在某些模型上实现了显著的加速和能耗降低，而没有精度损失。另一方面，ELSA重新构建了注意力机制，以在实数运算中保留精确的softmax语义，在各种平台和精度上提供硬件无关的性能提升和内存减少。 AI
影响 新的注意力算法提供了显著的加速和内存效率，可能降低推理成本并支持在资源受限设备上的部署。
排序理由 两篇学术论文介绍了优化vision transformer中注意力机制的新算法方法。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2604.25306v1 Announce Type: new Abstract: FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult. We identify three main obstacles to integer-only FlashA…
FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult. We identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise a…
arXiv:2604.23798v1 Announce Type: cross Abstract: Existing attention accelerators often trade exact softmax semantics, depend on fused Tensor Core kernels, or incur sequential depth that limits FP32 throughput on long sequences. We present \textbf{ELSA}, an algorithmic reformulat…