New PORTAL method skips projector training for multimodal LLMs

Researchers have developed a new method called PORTAL for training multimodal large language models (LLMs) that bypasses the traditional projector training stage. This new approach uses summary statistics from each modality to create an optimal-transport-based map, effectively skipping the need for paired image-caption data and gradient-based optimization. PORTAL demonstrates strong performance, matching standard pipelines trained on extensive datasets across various LLM and vision encoder configurations, and even outperforming them in some settings. AI

IMPACT This research could streamline the training process for multimodal AI systems, potentially reducing data and computational requirements.

RANK_REASON The cluster contains an academic paper detailing a new method for training multimodal LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Hugging Face
• image-caption datasets
• language model
• Multimodal LLMs
• vision encoder

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →