Beyond Self-Attention: Sub-Quadratic Vision Transformers for Fast Image Captioning
Researchers have developed a new vision transformer architecture that significantly reduces computational costs for image captioning. By replacing the standard self-attention mechanism with a Gaussian Mixture Model-based clustering approach, the model groups similar image patches, lowering complexity from quadratic to linear. This method, utilizing an Expectation-Maximization algorithm and a GPT-based decoder, achieves competitive results on the Flickr 30K dataset. AI
IMPACT Reduces computational overhead for image captioning models, potentially enabling faster and more efficient applications.