New Quat-Sphere-Vision model uses single quaternion for attention

Researchers have introduced Quat-Sphere-Vision (QSV), a novel sparse spherical vision model that utilizes a single learned unit quaternion per token for attention mechanisms. This approach replaces the traditional three learned projections (W_Q, W_K, W_V) with a single quaternion that handles both attention logits and feature transport. Experiments on CIFAR-10 and CIFAR-100 datasets showed that removing the feature transport component reduced test accuracy by approximately four percentage points, while replacing learned attention weights with uniform averaging had a negligible impact. AI

IMPACT Introduces a novel approach to attention mechanisms in vision models, potentially improving efficiency and performance.

RANK_REASON The cluster contains an academic paper detailing a new model architecture and experimental results. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• CIFAR-10
• CIFAR-100
• Quat-Sphere-Vision
• ResNet-20

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →