研究人员引入了一个名为Mixture of Probes (MoP)的新框架，旨在增强多模态大型语言模型 (MLLMs)。MoP解决了辅助模态（仅在训练期间可用）在现有MLLMs中常常被利用不足的挑战。该框架将模态特定信号和通用信号解耦，使模型能够学习可迁移的表示，并利用来自这些特权模态的互补监督。MoP在各种任务和模态上持续优于基线MLLMs，性能相对提升高达65%。 AI
影响 该框架可以通过使多模态AI系统更好地利用所有可用的训练数据来提高其性能，即使某些数据在推理时不可用。
排序理由 该集群包含一篇学术论文，详细介绍了多模态LLM的新框架和训练策略。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2503.08884v3 Announce Type: replace-cross Abstract: Unimodal vision models are known to rely on spurious correlations, but it remains unclear to what extent Multimodal Large Language Models (MLLMs) exhibit similar biases despite language supervision. In this paper, we inves…
arXiv:2607.08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, re…
Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modal…