Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
Researchers have developed CoMET, a novel method for multimodal classification that leverages frozen pre-trained backbones and Tabular Foundation Models (TFMs). This approach uses Principal Component Analysis (PCA) to compress modality embeddings before feeding them into a TFM, eliminating the need for fine-tuning. For improved representation quality, especially when CLS tokens are misaligned, they propose PALPooling, an adaptive token pooler. CoMET achieves state-of-the-art results on various multimodal benchmarks and can handle large-scale datasets with over 500,000 samples and 2,000 classes without any training. AI
IMPACT This method challenges traditional fine-tuning approaches, potentially enabling faster and more scalable multimodal classification across various domains.