Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning
A new research paper proposes that feature alignment, rather than data scale, is the key factor in choosing between cross-attention and concatenation for multimodal fusion. The study demonstrates that when features are pre-aligned through vision-language pretraining, concatenation outperforms cross-attention by a significant margin across various dataset sizes. This finding is supported by a theoretical analysis showing concatenation's superior sample efficiency, offering a principled framework for designing multimodal large language models. AI
IMPACT Provides a principled framework for selecting fusion methods in multimodal AI, potentially improving the design of LLMs.