New Multi-Overlapped-Head Self-Attention boosts Vision Transformer performance

Researchers have introduced Multi-Overlapped-Head Self-Attention (MOHSA), a novel mechanism designed to enhance Vision Transformers. Unlike standard Multi-Head Self-Attention (MHSA) which isolates attention heads, MOHSA allows for soft, overlapping divisions of queries, keys, and values between adjacent heads. This inter-head communication within the attention computation leads to improved feature representations. Experiments show that MOHSA provides a performance boost on several benchmark datasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1k, with only a minor increase in computational cost. AI

IMPACT Introduces a new attention mechanism that could improve the efficiency and performance of vision transformer models.

RANK_REASON The cluster describes a novel mechanism proposed in a research paper submitted to arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• CIFAR-10
• CIFAR-100
• ImageNet-1k
• Multi-Overlapped-Head Self-Attention
• Tianxiao Zhang
• Tiny-ImageNet
• Vision Transformers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →